acceptodds
Under review as a conference paper at ICLR 2027

Grounding Force: Image-Aligned Contact and Force Prediction for Vision-Language-Action Models

Abstract

Vision-language-action (VLA) models inherit semantic understanding from pretrained vision-language models and map observations and instructions directly to robot actions. The success of an action, however, also depends on the physical interaction, where the robot makes contact with an object and how it applies force. A VLA trained only to imitate actions never represents this physical interaction explicitly. Prior work trains the VLA to predict targets that describe the scene rather than the interaction, or measures the interaction with force or tactile sensors, which most robots and datasets lack. We present Grounding Force (GF), which trains a VLA to predict, alongside its next action, the contact and force that action will produce, as a contact map and force maps on the wrist-camera image. This supervision is used only in training, so the policy needs no force or tactile sensor at test time. Across four backbones (, , GR00T N1.6, and OpenVLA-OFT) on RoboCasa and RoboTwin 2.0, Grounding Force consistently improves the base VLAs, by up to 7.3% in success rate, and helps more than predicting future images, depth, or semantics. With pseudo-labels from a labeler trained only in simulation, it also improves by 13.0% on both real single-arm and bimanual robots. These results indicate that predicting the physical interaction in the image frame, without any sensor at test time, is an effective supervision signal for VLAs. Code and videos: https://groundingforce.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robotic Manipulation, Contact and Force Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QKbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Grounding Force (GF), an auxiliary training objective for VLA models. GF uses simulation-derived labels to train the model to predict a contact map and three signed force maps aligned with the wrist-camera image, alongside its next action. For real-robot training, the authors generate pseudo-labels using a model trained in simulation. The map decoder is removed at inference, so the policy requires no force or tactile sensor. Experiments across four VLA backbones and two simulation benchmarks report improvements in average task success; experiments report gains on ten real-robot tasks.

Critical strengths
  1. The paper addresses an important limitation of action imitation, where small motion errors can lead to consequential differences in contact and force. Aligning the auxiliary targets with the wrist-camera image is a clear and intuitive design choice.

  2. The evaluation covers multiple VLA architectures, single-arm and bimanual tasks, and both simulation and real robots. The ablations help identify which parts of GF contribute to the reported gains.

Critical weaknesses
  1. The reliability of the proposed method is highly questionable. Without input from external tactile sensors or other modalities, how can visual observations (multi-view RGB images) alone support accurate prediction of contact-force magnitudes? Moreover, the simulation data on which the authors rely lack stiffness properties: the gripper does not deform when different forces are applied.

  2. How to scale the data is a very important question. The proposed method seems applicable only to data re-generated in simulation. How could existing large-scale real-world datasets, egocentric datasets, and simulation datasets be incorporated into this method?

  3. The authors should report the additional training cost introduced by GF, as this is necessary to assess whether the method is worthwhile.

Questions for authors

See Weaknesses 1 and 2.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QKbw (1/2)

Thank you for raising these important questions, and for recognizing that aligning the targets with the wrist-camera image is a clear and intuitive design. We agree with the limitations you raise, and below we describe what our current results show, how we revised the paper, and how we plan to extend GF.

[W1] How can RGB observations alone support accurate prediction of contact-force magnitudes? The simulation lacks stiffness, and the gripper does not deform.

We agree that predicting accurate contact-force magnitudes from RGB observations alone is challenging. GF aims to predict from these observations where contact will occur at the next step and the force that will act there, as maps aligned with the wrist-camera image. To measure how well GF does this, we compared its predicted maps with the simulator labels on the 109 held-out RoboCasa demonstrations, which are not used in training. The table below repeats the main results of our reply to Reviewer f3yq (Q2-b) for π0.5 with GF, and Appendix G gives the full protocol.

Metricπ0.5 + GFReference without prediction
Contact location error (px on the 128 × 128 image; one patch = 8 px) ↓3.717.9 (fixed pixel at the mean training contact)
Force direction cosine ↑0.800.02 (mean training direction)
Force magnitude error (normalized label space) ↓0.340.87 (zero force)
Force magnitude correlation (Pearson r) ↑0.49–

These results indicate that GF accurately predicts where contact occurs and in which direction the force acts, and that its predicted magnitudes follow the labels to a moderate degree. Since the labels come from the simulator, the predicted magnitudes are in the normalized label space rather than in newtons, which we state as a limitation in the revised paper (Section 5).

We also agree that the simulators model rigid bodies and do not model gripper compliance, so the gripper does not deform under different forces. Tactile or force–torque sensing is the most direct way to obtain accurate force magnitudes, and incorporating such sensing into GF, together with simulators that model compliant grippers, is a main goal of our future work (Section 5).

AuthorsOpenReview ↗

Response to Reviewer QKbw (2/2)

[W2] How could existing large-scale real-world, egocentric and simulation datasets be incorporated?

We agree that scaling the data is a central question, and the current paper does not yet show GF on large-scale or egocentric datasets. Below we describe how each type of data can be used with GF and where GF is currently limited.

  • Existing simulation datasets: Labels come from replaying the recorded states. For RoboCasa we labeled the benchmark's existing human demonstrations this way, without collecting new trajectories. The same replay applies to large datasets generated automatically in recent simulators, as long as they store the simulator states.
  • Existing real datasets: Existing real datasets are especially important for scaling GF. As a first step, our real-robot experiments used a labeler trained on simulated RoboCasa demonstrations to pseudo-label real demonstrations recorded without force or tactile sensors, and GF trained with these labels improved π0.5 on all ten tasks. Because the labeler needs only wrist-camera images, it can be applied to other real datasets in the same way. Its pseudo-labels currently mark the grasp at the fingertips (Section 5), and we plan to explore real-to-sim-to-real pipelines that rebuild real scenes in simulation to provide richer labels.
  • Egocentric human videos: GF does not yet apply to egocentric human videos, and we consider this a limitation. The labeler is trained on gripper contacts in simulation, whereas these videos show human hands. Methods that reconstruct human videos in simulation could make labeling them possible, which we leave to future work (Appendix A).

[W3] What is the additional training cost of GF?

Thank you for this suggestion. GF trains with the same batch size and number of steps as each base model (Table 4), and the labels are computed once, offline. To measure the additional cost, we trained each base model and its GF version alternately on the same B200 GPU, with the per-GPU batch size of the RoboCasa runs, and timed 150 training steps of each after warm-up, excluding data loading.

ModelTime per step (s), baseTime per step (s), + GFPeak GPU memory (GiB), basePeak GPU memory (GiB), + GF
π00.6730.677 (+0.7%)67.171.7 (+6.7%)
π0.50.7850.864 (+10.1%)70.775.8 (+7.2%)
GR00T N1.60.9580.983 (+2.6%)138.9144.1 (+3.7%)
OpenVLA-OFT0.6300.638 (+1.4%)47.848.6 (+1.8%)

GF adds at most 10% to the time per training step and 2–7% to the peak GPU memory. At inference, the decoder is removed and the overhead is 0.38 ms per action chunk (1.1%, Table 5). We add these measurements to Appendix D.

Your comments helped us state the scope of GF more precisely, and we will continue to develop it in the directions you suggested. Thank you once again for your valuable feedback and time. We hope these clarifications and new results address your concerns, and we would be grateful if you would consider them in your assessment.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer QKbw,

Thank you again for your thoughtful questions. As the discussion period is drawing to a close, we kindly ask you to review our responses and let us know whether they address your concerns. Following your review, we added the contact and force prediction accuracy on held-out demonstrations (Appendix G), clarified that GF predicts force in the normalized label space and stated this as a limitation with tactile or force–torque sensing as future work (Section 5), discussed how GF can use existing datasets and its current limitation for egocentric videos (Appendix A), and reported the training cost of GF (Appendix D). We would be happy to answer any remaining questions.

Best regards,

Authors

Reviewer_nfbaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Grounding Force (GF) introduces contact and force prediction as training-time auxiliary supervision for vision-language-action models. Simulator contacts and signed force components are represented as maps in the wrist-camera image. Nine learnable grounding queries in the VLM provide features to both a map decoder and the action head. The map decoder is discarded at deployment, and no force or tactile sensor is required. For real demonstrations, a labeler trained on RoboCasa supplies pseudo-labels. Evaluation covers 24 RoboCasa tasks, 18 bimanual RoboTwin 2.0 tasks, and 10 real-robot tasks. Across the reported backbone/benchmark combinations, GF improves average success by 2.17–7.26 percentage points on RoboCasa and 1.34–5.39 points on RoboTwin. The reported real-robot improvement is 13.0 points for both single-arm and bimanual setups. Ablations examine prediction targets, force representations, query placement, and query count.

Critical strengths
  1. A practical auxiliary target with little deployment overhead. Image-aligned contact and signed force maps give the policy a compact interaction prediction objective that can be attached to several action architectures. Table 5 reports only 1.1% additional inference time for π0. The ablations provide useful evidence beyond the effect of adding tokens: contact alone gives no improvement in Fig. 3, whereas contact plus force improves success by 5.31 points; retaining separate force components also outperforms a merged force map in Fig. 4. These findings make the choice of supervision a meaningful contribution.

  2. Consistent average improvements across architectures and real-robot settings. The reported averages improve for four backbones on RoboCasa and three on RoboTwin. Simulation-generated pseudo-labels also improve all ten real tasks, with a 13.0-point average gain in each robot setup. This breadth supports the method's practical value despite the limitations in label quality and evaluation scope. The paper also clearly acknowledges that the real pseudo-labels mainly capture fingertip grasping rather than held-object/environment contact.

Critical weaknesses
  1. The real-robot evidence does not yet isolate the contribution of force information. Section 4.4 and Appendix H report that pseudo-labels remain at the fingertips and miss contacts between held objects and the environment. Figure 16 evaluates agreement with gripper opening/closing, not force direction or magnitude. Since the labeler is a trained π0.5+GF model, the gains could partly reflect teacher-derived auxiliary supervision or a useful grasp representation. The simulation contact-only ablation is informative, but does not resolve this question for real pseudo-labels. Real-robot contact-only and simple grasp-force-template controls would help establish whether the predicted force values add value beyond these alternatives. A related attribution issue in Fig. 3 is that the alternative future prediction targets use the third-person view, whereas GF uses the wrist view; target content and viewpoint therefore change together.

  2. Statistical reliability and generalization remain incompletely established. Appendix E states that RoboCasa evaluation includes the initial states of training demonstrations, so these results primarily establish in-distribution control improvements. RoboTwin uses held-out scenes, but only in the clean, seen-instruction setting. The paper does not report variability across independent training seeds or confidence intervals for success-rate differences, leaving the stability of smaller gains, such as the 1.34-point improvement for π0 on RoboTwin, uncertain. Moreover, the π0.5 RoboTwin average is substantially influenced by stack_blocks_two improving from 8% to 74%; this task alone contributes about 3.67 points to the 18-task average gain. The cross-backbone and real-robot results still support a weak acceptance recommendation, but broader robustness and generalization claims would require stronger evidence.

Questions for authors

The following questions concern evidence or clarifications that could affect the assessment, rather than suggestions solely for future work.

  1. Do real-robot gains persist against matched contact-only pseudo-labels and contact maps paired with fixed inward grasp-force templates? If available, a matched teacher-feature distillation control would also be informative. The central question is whether the learned force directions and magnitudes improve success beyond grasp localization or generic teacher supervision.

  2. Does GF retain its advantage over future-image, SigLIP, depth, and semantic targets when those targets also predict the wrist view, with the same queries/decoder and comparable tuning budgets? The current third-person controls do not isolate the contribution of physical target content.

  3. How many independent training seeds were used for the main comparisons and key ablations? Please report uncertainty in the success-rate differences and, if available, RoboCasa results strictly excluding training initial states. These would affect confidence in the smaller gains and the scope of generalization.

  4. Please clarify the coordinate convention for Y_(t+1): are contact positions and forces at t+1 projected and rotated using the wrist-camera pose at t or t+1? The target is described as aligned with the current observation I_t^w, while Appendix A refers to the camera pose at that step. Also, since the predictor does not receive a candidate action, should its output be interpreted as the next interaction under the demonstration policy, rather than the interaction caused by the generated action? These distinctions matter for reproducibility and interpretation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nfba

Thank you for the thorough review and the precise questions, and for recognizing GF as a practical auxiliary target with little deployment overhead. We respond to each below.

[W1 / Q1] Do real-robot gains persist against matched contact-only pseudo-labels and fixed inward grasp-force templates?

Thank you for suggesting these comparisons. As described in General Response 3, with the same π0.5 backbone, data and recipe on the bimanual setup, the average success rates are 76.0 (base), 79.5 (contact-only pseudo-labels), 84.0 (contact + fixed inward force) and 89.0 (GF), with the same order on all four tasks. GF therefore improves on the fixed-inward-force variant by 5.0 points and on the contact-only variant by 9.5 points. Because the contact-only variant uses the contact maps of the same labeler, the latter gap reflects the learned force rather than generic supervision from the labeler. We also appreciate the suggestion of a teacher-feature distillation variant. Within the discussion period we focused on these two variants, because they directly separate the contribution of the predicted force from that of grasp supervision.

[W1 / Q2] Does GF keep its advantage when the future targets also predict the wrist view, with the same queries and decoder?

Thank you for this suggestion. We retrained all four future-frame targets of Fig. 3 to predict the next wrist-camera frame, with the same nine queries, the same decoder and the same training recipe as GF. Each target was trained three times, and each run was evaluated once. GF remains the best target on the 24 RoboCasa tasks: 58.57 ± 0.52 (+4.08 over the base model), against 57.09 ± 0.79 for semantics, 56.93 ± 0.63 for SigLIP features, 56.47 ± 1.21 for depth and 54.05 ± 0.72 for pixels. On the 109 held-out initial states its lead is larger still: +12.30 over the base model, against +2.24 to +7.40 for the other targets. The full table is in our response to Reviewer f3yq, Q3-a.

[W2 / Q3] How many independent training seeds were used? Please report uncertainty, and RoboCasa results that exclude training initial states.

Thank you for this question. Please see General Response 2. In the submission, each result came from one training run evaluated once. Following your suggestion, we trained each model two more times and now report three independent training runs, each evaluated once, for all seven backbone–benchmark pairs, as mean ± std, with a bootstrap 95% CI over tasks for every gain in Appendix F. The submitted run is one of the three, and GF is higher than the base model in every run. The wrist-view targets in our reply to Q2 are also trained three times each, with 95% CIs in Appendix I.1. For the smaller gain you mention, π0 on RoboTwin, the three runs give +1.34, +1.17 and +3.78. The gain is positive in every run, and its mean of +2.09 has a 95% CI of [+0.02, +4.13], which lies above zero.

On the 109 RoboCasa episodes whose initial states are not in the training demonstrations, GF improves all four backbones: π0 +12.30, π0.5 +4.43, GR00T N1.6 +7.63, OpenVLA-OFT +6.13.

As you note, stack_blocks_two accounts for a large part of the π0.5 gain on RoboTwin. When we exclude this task, GF still improves π0.5 by 2.5 points on the other 17 tasks (66.33 → 68.84, 95% CI [+0.73, +4.96]). The gain on stack_blocks_two itself was consistent across the three runs (base 8, 6 and 4%; GF 74, 73 and 76%). On RoboTwin we evaluate held-out scenes in the clean, seen-instruction setting, which the revised paper now states in Section 4.1.

[Q4] Are Yt+1Y_{t+1} contacts and forces projected with the wrist-camera pose at t or at t+1? Is the output the next interaction under the demonstration policy?

Thank you for the careful reading. In simulation, we project the contact positions at t+1 and rotate the force vectors with the wrist-camera pose at t+1. The policy observes ItI_t, and the target is the map in the wrist-camera frame of step t+1. The last frame of an episode uses its own label. On the real robot there is no projection, since the labeler predicts the maps directly from each wrist image. We apologize for describing the target as aligned with the current observation ItI_t, which was imprecise, and we clarify this in the revised paper (Sections 3.1 and 3.2 and Appendix A).

Your interpretation is correct. Because the labels come from the recorded demonstration states, the target is the next interaction under the demonstration policy, not the interaction that the generated action would cause. The grounding queries do not receive the action being generated, and the decoder is not used at test time. We state this in the revised paper (Appendix A).

Thank you once again for your valuable feedback and time. We hope these results address your concerns, and we are happy to answer further questions.

Thanks. I remain positive about the acceptance of this paper.

Thank you for your constructive suggestions, which directly strengthened our work, and we appreciate your continued positive assessment! Please let us know if you have any further questions.

Reviewer_f3yqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Grounding Force (GF), an auxiliary supervision method for vision-language-action (VLA) models. GF trains learnable query tokens to predict next-step contact locations and signed force components as maps aligned with the wrist-camera image, and the action head also reads these query representations. Contact and force labels come from simulation; for real demonstrations, a simulation-trained model supplies pseudo-labels. At deployment, the map decoder is discarded, leaving only nine additional query tokens and requiring no force or tactile measurements.

The evaluation covers four VLA backbones on RoboCasa, three on RoboTwin 2.0, and ten real-world tasks using π₀.₅. GF improves average success in every reported backbone-benchmark comparison and yields a reported average improvement of 13 percentage points on both the real single-arm and bimanual setups. Ablations examine prediction targets, force representations, query placement, and query count.

Critical strengths

The method offers a practical way to incorporate interaction supervision into existing VLAs. The image-aligned contact and force targets provide spatially localized information about interaction without requiring additional sensor measurements at deployment. The query-token design accommodates substantially different action heads, and the decoder can be removed after training. The measured inference overhead is only 1.1% for π₀, which makes the approach attractive as an addition to existing policies.

The empirical evaluation is broad and includes meaningful real-world evidence. Average gains appear across all seven reported backbone-benchmark combinations, including one with a strong π₀.₅ baseline. The real-world experiments cover ten tasks with 50 evaluations per task, and average success improves by 13 percentage points on both robot setups. Matching training recipes and evaluation checkpoints strengthens the base-versus-GF comparisons, and the OpenVLA-OFT checkpoint study reduces concern that its gain depends on a favorable checkpoint. Together, these results support the usefulness of the auxiliary supervision beyond one architecture or simulator.

Critical weaknesses

The statistical reliability and generalization scope are insufficiently characterized. The manuscript does not report the number of independent training runs or provide uncertainty estimates for success rates and ablation differences. This matters especially for smaller gains, such as π₀’s 1.34-point improvement on RoboTwin, and for interpreting differences among auxiliary targets. Appendix E also states that RoboCasa evaluation includes the initial states of training demonstrations. Even though RoboTwin uses held-out scenes and the real trials randomize object placement, the principal simulation ablations therefore establish benefits only in a relatively restricted evaluation setting. The checkpoint study is useful but does not establish robustness across training seeds.

The contribution of force information is not fully separated from other forms of auxiliary supervision. On real robots, the pseudo-labels reportedly stay at the fingertips and do not identify held-object-environment contacts. Their quantitative validation measures agreement with gripper closure rather than force direction or magnitude. The real-world gains could therefore partly arise from grasp-related supervision or knowledge transferred from the simulation-trained labeler, and the contact-only ablation on simulated π₀ does not resolve this question for real π₀.₅. In addition, future-frame baselines predict the third-person view while GF predicts wrist-view maps, and the image-versus-world comparison changes both the coordinate frame and the output representation. These comparisons support the full GF design but do not cleanly isolate the claimed advantages of force content and image alignment.

Questions for authors

Statistical reliability: How many independent training seeds were used for the main comparisons and ablations, and could you provide uncertainty estimates for the key gains? Could you also clarify whether the RoboCasa improvements hold on initial states excluded from the training demonstrations?

Force versus grasp supervision: Do the real-world gains persist when the pseudo-labels are replaced by simpler grasp-related targets, such as fingertip contact maps with fixed inward force directions? Reporting quantitative contact-localization and force-prediction results on held-out simulation data would also help show what physical information the queries learn.

Baselines and coordinate frame: Does GF still outperform future-image, feature, or depth supervision when those targets are defined in the same wrist-camera view? For the coordinate-frame claim, could you separate the effect of the frame from the effect of the output representation (dense maps versus 3D point/vector outputs)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer f3yq (1/3)

Thank you for the detailed and constructive review, and for recognizing the practicality of GF and the breadth of our evaluation. We respond to each weakness and question below, and tag each item with the weakness (W1, W2) and the question it addresses.

[W1 / Q1-a] How many independent training seeds were used, and what is the uncertainty of the key gains?

Thank you for this question. Please see General Response 2. In the submission, each result came from one training run evaluated once. Following your suggestion, we trained each model two more times and now report three independent training runs, each evaluated once, for all seven backbone–benchmark pairs. The submitted run is one of the three, and GF is higher than the base model in every run. The new ablations in our replies to Q3-a and Q3-b also use three training runs each, with 95% CIs in Appendices I.1 and I.2. Together with the OpenVLA-OFT checkpoint study in the submission, the three runs show that the gain holds across both checkpoints and training seeds. On the smaller gain you mention, π0 on RoboTwin, the three runs give +1.34, +1.17 and +3.78. The gain is positive in every run, and its mean of +2.09 has a 95% CI of [+0.02, +4.13], which lies above zero.

[W1 / Q1-b] Do the RoboCasa improvements hold on initial states excluded from the training demonstrations?

Thank you for raising this point. As shown in General Response 2, on the 109 RoboCasa episodes whose initial states are not in the training demonstrations, GF improves all four backbones: π0 +12.30, π0.5 +4.43, GR00T N1.6 +7.63, OpenVLA-OFT +6.13. For π0 and GR00T N1.6 the gain is larger than on the full set.

[W2 / Q2-a] Do the real-world gains persist against simpler grasp-related targets, such as fingertip contact maps with fixed inward force directions?

Thank you for suggesting this comparison. As described in General Response 3, we trained the variant you describe, fingertip contacts with a fixed inward force, as well as a contact-only variant, with the same π0.5 backbone, data and recipe. The average success rates over four bimanual tasks are 76.0 (base), 79.5 (contact only), 84.0 (fixed inward force) and 89.0 (GF), with the same order on every task. GF improves on the fixed-inward-force variant by 5.0 points and on the contact-only variant by 9.5 points. Because the contact-only variant uses the contact maps of the same labeler, the latter gap reflects the predicted force rather than knowledge transferred from the labeler in general.

AuthorsOpenReview ↗

Response to Reviewer f3yq (2/3)

[W2 / Q2-b] Quantitative contact localization and force prediction on held-out simulation data.

Thank you for this suggestion. We agree that a direct evaluation of the predicted maps is useful. We evaluate the maps predicted by the GF decoder on the 109 held-out RoboCasa demonstrations, which are not used in training, against ground-truth maps produced by the same code as the training labels. We use one of every four time steps of each demonstration (7,955 of 31,643 frames, 2,972 with contact), and using all frames gives nearly the same results. We report π0.5 + GF, the backbone of our real-robot experiments. Each metric is compared with a reference that uses no prediction, and ↑ (↓) marks metrics for which higher (lower) is better. Appendix G gives the full protocol and the results for π0 and GR00T N1.6.

MetricWhat it measuresπ0.5 + GFReference without prediction
Contact location error (px) ↓Median distance from the predicted contact to the nearest true contact on the 128 × 128 wrist image (one patch = 8 px)3.717.9 (fixed pixel at the mean training contact)
Contact within one patch ↑Fraction of contact frames whose predicted contact lies within 8 px of a true contact (0 to 1)0.800.14 (same fixed pixel)
Contact-frame detection accuracy ↑Fraction of frames for which the decoder correctly predicts whether contact occurs (0 to 1)0.950.63 (always "no contact")
Force direction cosine ↑Cosine between the predicted and true force vectors in contact cells (−1 to 1; 1 = same direction, 0 = unrelated)0.800.02 (mean training direction)
Force magnitude error ↓Mean absolute error of the force magnitude in contact cells, in the normalized label space0.340.87 (zero force)
Force magnitude correlation ↑Pearson r between the predicted and true magnitudes in contact cells (−1 to 1; 0 = unrelated)0.49–

The grounding queries therefore encode where contact will occur, with a median error below half a patch, and in which direction force will act, with a cosine of 0.80 to the ground truth. The predicted magnitude also follows the label (r = 0.49), with an error 61% lower than that of a zero prediction. π0 and GR00T N1.6 give similar values (Appendix G). In Fig. 3, a contact map alone gives no gain (−0.09), while contact with force gives +5.31, which indicates that the force information improves the policy. We add this evaluation to the revised paper (Appendix G).

[W2 / Q3-a] Does GF still outperform future-image, feature or depth supervision when those targets use the same wrist view?

Thank you for pointing out that target content and viewpoint change together in Fig. 3. We retrained all four future-frame targets of Fig. 3 to predict the next wrist-camera frame instead of the next third-person frame. They use the same nine queries, the same decoder and the same training recipe as GF. Each target was trained three times, and each run was evaluated once (95% CIs in Appendix I.1).

Target on the next wrist frameRoboCasa, 24 tasks (change from base)RoboCasa held-out, 109 episodes (change from base)
None (base)54.49 ± 0.6337.53 ± 6.82
Pixels54.05 ± 0.72 (−0.44)42.80 ± 1.37 (+5.27)
SigLIP features56.93 ± 0.63 (+2.44)39.77 ± 3.69 (+2.24)
Depth56.47 ± 1.21 (+1.98)41.60 ± 2.99 (+4.07)
Semantics57.09 ± 0.79 (+2.60)44.93 ± 1.62 (+7.40)
Contact + force (GF)58.57 ± 0.52 (+4.08)49.83 ± 2.32 (+12.30)

GF remains the best target. Its gain over the base model (+4.08) is about 1.6 times that of the best alternative, semantics (+2.60), the same ratio as in Fig. 3 of the submission. On the held-out initial states the lead of GF is larger still: +12.30 over the base model, against +2.24 to +7.40 for the other targets. GF is thus the most effective target whether the other targets predict the third-person view (Fig. 3) or the wrist view, which suggests that its advantage comes from predicting contact and force itself.

AuthorsOpenReview ↗

Response to Reviewer f3yq (3/3)

[W2 / Q3-b] Can the effect of the coordinate frame be separated from the effect of the output representation?

Thank you for this insightful question. We agree that in Fig. 4a of the submission, the world-frame variant changes both factors at once: GF predicts dense maps in the image frame of the wrist camera, while that variant predicts contact points and force vectors in the world frame. To separate the two effects, we trained the two variants that change only one of them, on π0 and RoboCasa:

  • Image frame, point set: GF's dense maps are replaced by a set of six predicted contacts. Each contact has an image position (u, v), an objectness score and a force vector, and the set is trained with Hungarian matching to the next-step contacts.
  • World frame, dense map: GF's 16 × 16 dense map is kept, but its grid is defined in world coordinates. The grid is a 24 cm × 24 cm horizontal window centered on the gripper, with 1.5 cm cells, and each cell holds contact, world-axis force and the contact's height offset. No camera projection is used.

Each variant was trained three times, and each run was evaluated once. Held-out results use the 109 excluded initial states.

VariantRoboCasa, 24 tasks (change from base)RoboCasa held-out, 109 episodes (change from base)
None (base)54.49 ± 0.6337.53 ± 6.82
Image frame, point set49.07 ± 0.31 (−5.42)38.50 ± 1.56 (+0.97)
World frame, dense map56.20 ± 0.53 (+1.71)41.87 ± 2.32 (+4.34)
World frame, point set (Fig. 4a, one run)52.72 (−1.77)39.45 (+1.92)
Image frame, dense map (GF)58.57 ± 0.52 (+4.08)49.83 ± 2.32 (+12.30)

Both the output representation and the coordinate frame contribute to the performance of GF. Replacing the dense maps with a point set in the image frame lowers the success rate by 9.50 points (95% CI [5.10, 14.19]), and defining the dense maps in the world frame instead of the image frame lowers it by 2.37 points ([0.48, 4.31]) on all episodes and by 7.96 points on the held-out states. GF, which predicts dense maps in the image frame, achieves the highest success rate on all episodes as well as on the held-out episodes. The two choices also reinforce each other, since dense maps improve the success rate far more in the image frame (+9.50 points) than in the world frame (+3.48 points relative to the single run of Fig. 4a). These results suggest that the gain of GF comes from predicting dense maps that are aligned with the image observed by the policy.

[Ethics flag] Regarding the ethics flag, our experiments use simulation benchmarks and our own lab robots, with no human subjects or personal data. If there is a specific concern, we would be glad to address it.

Thank you once again for your valuable feedback and time. We hope these results address your concerns, and we are happy to answer further questions.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer f3yq,

Thank you again for your detailed and constructive review. As the discussion period is drawing to a close, we kindly ask you to review our responses and let us know whether they address your concerns. Following your suggestions, we added three training runs with uncertainty estimates for all seven backbone–benchmark pairs, results on the 109 RoboCasa initial states excluded from training, the real-robot contact-only and fixed inward force variants, the contact and force prediction accuracy on held-out demonstrations, and the wrist-view target and coordinate-frame studies (General Responses 2 and 3; Appendices E–I of the revised PDF). We would be happy to answer any remaining questions.

Best regards,

Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response 1: summary of new results and revisions

We thank all reviewers for their careful and constructive reviews. We are encouraged that reviewers found Grounding Force (GF) a practical training-time target with negligible deployment overhead [f3yq, nfba], compatible with different action heads [f3yq, nfba], built on an intuitive image-aligned design [QKbw], and evaluated broadly across backbones, simulators and real robots [f3yq, nfba, QKbw].

New experiments and analyses (success rates in %):

  • Three training runs for all seven backbone–benchmark pairs [f3yq, nfba]: mean ± std and bootstrap 95% CIs over tasks. GF is higher than the base model in every run. → General Response 2, Tables 1–2 and Appendix F.
  • RoboCasa initial states excluded from training (109 episodes) [f3yq, nfba]: GF improves all four backbones, by 4.4 to 12.3 points. → General Response 2, Appendix E.
  • Real-robot force vs. grasp supervision [f3yq, nfba]: on the bimanual setup with π0.5, base 76.0, contact only 79.5, contact + fixed inward force 84.0, GF 89.0, with the same order on all four tasks. → General Response 3, Appendix H.
  • Future-frame targets on the same wrist view [f3yq, nfba]: GF remains the best target (58.57 vs. at most 57.09). → reply to f3yq Q3-a, Appendix I.1.
  • Coordinate frame vs. output representation [f3yq]: both choices contribute. Replacing the dense maps with a point set lowers the success rate by 9.5 points, and defining the maps in the world frame lowers it by 2.4 points (8.0 on the held-out states). → reply to f3yq Q3-b, Appendix I.2.
  • Contact and force prediction accuracy on held-out demonstrations [f3yq, QKbw]: median contact location error below half a patch (3.7 px), force direction cosine 0.80, force magnitude correlated with the label (r = 0.49). → reply to f3yq Q2-b, Appendix G.
  • Training cost [QKbw]: GF adds at most 10% to the time per training step and 2–7% to the peak GPU memory. → reply to QKbw W3, Appendix D.

Revisions to the paper (marked in blue in the revised PDF):

  • Tables 1 and 2 report three-run means ± std, with every run in Appendix F and per-task results in Appendix E.
  • The other new results are added to the existing appendices they relate to, with the training cost in Appendix D, the held-out evaluation in Appendix E, the prediction accuracy in Appendix G and the real-robot grasp-supervision variants in Appendix H. A new Appendix I contains the wrist-view targets and the study of the coordinate frame and the output representation.
  • Clarifications: Yt+1Y_{t+1} uses the wrist-camera pose at t+1 and is the next interaction under the demonstration policy [nfba]. RoboTwin is evaluated in the clean, seen-instruction setting, which we now state in Section 4.1 [nfba]. GF predicts the contact location and the direction and magnitude of force in the normalized label space, not calibrated force values [QKbw]. We also discuss the application of GF to existing datasets, including its current limitation for egocentric videos (Appendix A), and add tactile or force–torque sensing as future work (Section 5) [QKbw].
AuthorsOpenReview ↗

General Response 2: multiple runs and held-out initial states (f3yq Q1, nfba Q3)

We thank both reviewers for asking how many training runs support the main comparisons and whether the RoboCasa gains hold on initial states excluded from training. In the submission, each result came from a single training run evaluated once. Following the reviewers' comments, we trained two more runs for every backbone–benchmark pair, and we now report three independent training runs (each evaluated once), of which the submitted run is one.

Regarding the second question, RoboCasa evaluates each policy from the initial states of its 1,293 released demonstrations, and 1,184 of these are also used for training. The remaining 109 episodes (22 tasks) start from initial states that the model never sees during training. The "RoboCasa held-out" column reports the success rate on these 109 episodes, and thus shows whether the gain carries over to unseen initial states.

We report the average success rate (%) as the mean ± standard deviation over the three runs. As in Tables 1 and 2 of the paper, the better of each base VLA and its GF version is shown in bold, and the number in parentheses is the gain from GF.

RoboCasa (24 tasks)RoboCasa held-out (109 episodes)RoboTwin 2.0 (18 tasks)
π054.49 ± 0.6337.53 ± 6.8249.87 ± 0.84
π0 + GF58.57 ± 0.52 (+4.08)49.83 ± 2.32 (+12.30)51.96 ± 1.37 (+2.09)
π0.565.39 ± 0.9958.27 ± 1.3562.98 ± 0.72
π0.5 + GF73.40 ± 0.36 (+8.01)62.70 ± 0.52 (+4.43)69.15 ± 0.12 (+6.17)
GR00T N1.655.47 ± 0.0941.90 ± 1.37–
GR00T N1.6 + GF57.92 ± 0.51 (+2.45)49.53 ± 4.88 (+7.63)–
OpenVLA-OFT30.51 ± 0.5922.30 ± 1.0424.37 ± 1.13
OpenVLA-OFT + GF36.80 ± 0.49 (+6.29)28.43 ± 1.85 (+6.13)27.96 ± 0.84 (+3.59)

In each of the three runs, GF is higher than the base model on all seven backbone–benchmark pairs, and Appendix F lists every run with a bootstrap confidence interval over tasks for each gain. On the held-out episodes, GF improves all four backbones, and for π0 and GR00T N1.6 the gain is larger than on the full set.

AuthorsOpenReview ↗

General Response 3: real-robot force vs. grasp supervision (f3yq Q2, nfba Q1)

We thank both reviewers for asking whether the real-robot gain comes from the predicted force itself or from simpler grasp supervision. Following their suggestion, we trained two additional variants on the bimanual setup, using the same π0.5 backbone, data and training recipe as GF:

  • Contact only: the labeler's contact map, with the force maps left unsupervised.
  • Contact + fixed inward force: fingertip contacts on closed-gripper frames with a fixed inward force on each fingertip, without any labeler output.

The base and GF results are those of Fig. 5b, and each model was evaluated 50 times per task.

Taskπ0.5 (base)Contact onlyContact + fixed inward forceπ0.5 + GF
Drawer and can80.082.086.092.0
Cup stacking84.088.092.094.0
Lego box64.070.078.086.0
Cube stacking76.078.080.084.0
Average76.079.584.089.0

GF achieves the highest success rate on all four tasks, followed by contact + fixed inward force, contact only and the base model. GF improves on the fixed-inward-force variant by 5.0 points and on the contact-only variant by 9.5 points. Because the contact-only variant is trained on the contact maps of the same labeler, the latter gap reflects the predicted force itself rather than supervision from the labeler in general.

AuthorsOpenReview ↗

General Response 4: summary of the discussion

Dear Area Chair and Reviewers,

We thank all reviewers and the AC for their time and constructive feedback. We are encouraged that reviewers found GF a practical training-time target with negligible deployment overhead [f3yq, nfba], built on a clear and intuitive image-aligned design [QKbw], and evaluated broadly across backbones, simulators and real robots [f3yq, nfba, QKbw]. As the discussion period comes to a close, we summarize the discussion below to support the final decision.

Status of the discussion. Reviewer nfba read our response and stated that they remain positive about the acceptance of the paper. Reviewers f3yq and QKbw have not replied yet, and we believe the responses summarized below address their concerns.

Concerns and new results. The table lists each concern, the new result and where it appears in the revised PDF (changes marked in blue). Success rates are in %, and gains in percentage points.

ReviewerConcernResultWhere
f3yq Q1-a, nfba Q3Number of training runs and uncertainty3 training runs for all 7 backbone–benchmark pairs. GF is higher than the base model in every run. π0 RoboTwin: +2.09, 95% CI [+0.02, +4.13]GR2, Tab. 1–2, App. F
f3yq Q1-b, nfba Q3RoboCasa initial states excluded from training109 held-out episodes: GF improves all 4 backbones by +4.4 to +12.3GR2, App. E
f3yq Q2-a, nfba Q1Simpler grasp targets on the real robotBimanual, π0.5: base 76.0 / contact only 79.5 / contact + fixed inward force 84.0 / GF 89.0, same order on all 4 tasksGR3, App. H
f3yq Q3-a, nfba Q2Future-frame targets on the same wrist viewGF remains the best target: 58.57 vs. at most 57.09 (3 runs each)App. I.1
f3yq Q3-bCoordinate frame vs. output representationRelative to GF, a point set instead of dense maps gives −9.5, and world-frame maps give −2.4 (−8.0 on held-out states)App. I.2
f3yq Q2-b, QKbw W1Contact and force prediction from RGBHeld-out demos: contact location error 3.7 px (below half a patch), force direction cosine 0.80, magnitude correlation with the label r = 0.49App. G
nfba Q3π0.5 RoboTwin gain without stack_blocks_two+2.5 on the other 17 tasks, 95% CI [+0.73, +4.96]Reply to nfba
QKbw W3Training and inference costAt most +10% time per training step, +2–7% peak GPU memory, and +0.38 ms (1.1%) per action chunk at inferenceApp. D

Clarifications and scope

  • [nfba Q4] Yt+1Y_{t+1} uses the wrist-camera pose at t+1, and the target is the next interaction under the demonstration policy (Sections 3.1–3.2, Appendix A).
  • [nfba Q3] RoboTwin is evaluated in the clean, seen-instruction setting (Section 4.1).
  • [QKbw W1] Force magnitudes are in the normalized label space, not in newtons. We state this as a limitation, with tactile or force–torque sensing and compliant-gripper simulation as future work (Section 5).
  • [QKbw W2] Simulation and real datasets can be labeled as described in Appendix A. Egocentric human videos remain a current limitation.
  • [f3yq, ethics flag] Our experiments use simulation benchmarks and our own lab robots, with no human subjects or personal data.

We kindly ask the reviewers to take the discussion above into consideration when finalizing their reviews, and we will gladly clarify any remaining point.

Sincerely,

Authors of Submission 53404

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.