- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Grounding Force (GF), an auxiliary training objective for VLA models. GF uses simulation-derived labels to train the model to predict a contact map and three signed force maps aligned with the wrist-camera image, alongside its next action. For real-robot training, the authors generate pseudo-labels using a model trained in simulation. The map decoder is removed at inference, so the policy requires no force or tactile sensor. Experiments across four VLA backbones and two simulation benchmarks report improvements in average task success; experiments report gains on ten real-robot tasks.
Critical strengths
-
The paper addresses an important limitation of action imitation, where small motion errors can lead to consequential differences in contact and force. Aligning the auxiliary targets with the wrist-camera image is a clear and intuitive design choice.
-
The evaluation covers multiple VLA architectures, single-arm and bimanual tasks, and both simulation and real robots. The ablations help identify which parts of GF contribute to the reported gains.
Critical weaknesses
-
The reliability of the proposed method is highly questionable. Without input from external tactile sensors or other modalities, how can visual observations (multi-view RGB images) alone support accurate prediction of contact-force magnitudes? Moreover, the simulation data on which the authors rely lack stiffness properties: the gripper does not deform when different forces are applied.
-
How to scale the data is a very important question. The proposed method seems applicable only to data re-generated in simulation. How could existing large-scale real-world datasets, egocentric datasets, and simulation datasets be incorporated into this method?
-
The authors should report the additional training cost introduced by GF, as this is necessary to assess whether the method is worthwhile.
Questions for authors
See Weaknesses 1 and 2.
3 comments
Response to Reviewer QKbw (2/2)
[W2] How could existing large-scale real-world, egocentric and simulation datasets be incorporated?
We agree that scaling the data is a central question, and the current paper does not yet show GF on large-scale or egocentric datasets. Below we describe how each type of data can be used with GF and where GF is currently limited.
- Existing simulation datasets: Labels come from replaying the recorded states. For RoboCasa we labeled the benchmark's existing human demonstrations this way, without collecting new trajectories. The same replay applies to large datasets generated automatically in recent simulators, as long as they store the simulator states.
- Existing real datasets: Existing real datasets are especially important for scaling GF. As a first step, our real-robot experiments used a labeler trained on simulated RoboCasa demonstrations to pseudo-label real demonstrations recorded without force or tactile sensors, and GF trained with these labels improved π0.5 on all ten tasks. Because the labeler needs only wrist-camera images, it can be applied to other real datasets in the same way. Its pseudo-labels currently mark the grasp at the fingertips (Section 5), and we plan to explore real-to-sim-to-real pipelines that rebuild real scenes in simulation to provide richer labels.
- Egocentric human videos: GF does not yet apply to egocentric human videos, and we consider this a limitation. The labeler is trained on gripper contacts in simulation, whereas these videos show human hands. Methods that reconstruct human videos in simulation could make labeling them possible, which we leave to future work (Appendix A).
[W3] What is the additional training cost of GF?
Thank you for this suggestion. GF trains with the same batch size and number of steps as each base model (Table 4), and the labels are computed once, offline. To measure the additional cost, we trained each base model and its GF version alternately on the same B200 GPU, with the per-GPU batch size of the RoboCasa runs, and timed 150 training steps of each after warm-up, excluding data loading.
| Model | Time per step (s), base | Time per step (s), + GF | Peak GPU memory (GiB), base | Peak GPU memory (GiB), + GF |
|---|---|---|---|---|
| π0 | 0.673 | 0.677 (+0.7%) | 67.1 | 71.7 (+6.7%) |
| π0.5 | 0.785 | 0.864 (+10.1%) | 70.7 | 75.8 (+7.2%) |
| GR00T N1.6 | 0.958 | 0.983 (+2.6%) | 138.9 | 144.1 (+3.7%) |
| OpenVLA-OFT | 0.630 | 0.638 (+1.4%) | 47.8 | 48.6 (+1.8%) |
GF adds at most 10% to the time per training step and 2–7% to the peak GPU memory. At inference, the decoder is removed and the overhead is 0.38 ms per action chunk (1.1%, Table 5). We add these measurements to Appendix D.
Your comments helped us state the scope of GF more precisely, and we will continue to develop it in the directions you suggested. Thank you once again for your valuable feedback and time. We hope these clarifications and new results address your concerns, and we would be grateful if you would consider them in your assessment.
Kind reminder
Dear Reviewer QKbw,
Thank you again for your thoughtful questions. As the discussion period is drawing to a close, we kindly ask you to review our responses and let us know whether they address your concerns. Following your review, we added the contact and force prediction accuracy on held-out demonstrations (Appendix G), clarified that GF predicts force in the normalized label space and stated this as a limitation with tactile or force–torque sensing as future work (Section 5), discussed how GF can use existing datasets and its current limitation for egocentric videos (Appendix A), and reported the training cost of GF (Appendix D). We would be happy to answer any remaining questions.
Best regards,
Authors
Response to Reviewer QKbw (1/2)
Thank you for raising these important questions, and for recognizing that aligning the targets with the wrist-camera image is a clear and intuitive design. We agree with the limitations you raise, and below we describe what our current results show, how we revised the paper, and how we plan to extend GF.
[W1] How can RGB observations alone support accurate prediction of contact-force magnitudes? The simulation lacks stiffness, and the gripper does not deform.
We agree that predicting accurate contact-force magnitudes from RGB observations alone is challenging. GF aims to predict from these observations where contact will occur at the next step and the force that will act there, as maps aligned with the wrist-camera image. To measure how well GF does this, we compared its predicted maps with the simulator labels on the 109 held-out RoboCasa demonstrations, which are not used in training. The table below repeats the main results of our reply to Reviewer f3yq (Q2-b) for π0.5 with GF, and Appendix G gives the full protocol.
These results indicate that GF accurately predicts where contact occurs and in which direction the force acts, and that its predicted magnitudes follow the labels to a moderate degree. Since the labels come from the simulator, the predicted magnitudes are in the normalized label space rather than in newtons, which we state as a limitation in the revised paper (Section 5).
We also agree that the simulators model rigid bodies and do not model gripper compliance, so the gripper does not deform under different forces. Tactile or force–torque sensing is the most direct way to obtain accurate force magnitudes, and incorporating such sensing into GF, together with simulators that model compliant grippers, is a main goal of our future work (Section 5).