acceptodds
Under review as a conference paper at ICLR 2027

Do Embodied AI Models Have Physical Intuition? An Egocentric Probing Benchmark

Abstract

Estimating contact forces from egocentric observations requires embodied models to map visual appearance and hand motion to continuous physical quantities. These estimates depend on how object geometry, deformation, and load evolve during interaction. Occlusion and visual–physical mismatch complicate this mapping across materials and fill states. We introduce FEEL (Force Estimation from Egocentric Learning), a controlled benchmark of 3 069 vision–touch–pose episodes in ten object and fill-state categories, and EgoForce, which uses a GRU for visual–state aggregation and two softly weighted residual experts to refine a shared Laplace force head. We evaluate transformer regression policies, flow-matching vision–language–action models, generalist multimodal policies, and vision-foundation encoders on current-force prediction from RGB and hand state. The 36.18 M-parameter EgoForce achieves test contact MAE of 7.3386 N, contact , contact Pearson , and pooled F1 of 0.8395, with OOD contact and pooled F1 of 0.8934 on empty milk containers. Project page: http://submission.anonymoussubmission.com/

Primary area:
datasets and benchmarks
Keywords:
Benchmarks and datasets for robot learning, Learning representations for robotic perception and control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5RD8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FEEL, an egocentric force-estimation benchmark containing 3,069 interaction episodes from four demonstrators across ten object/load conditions. Models use RGB observations and a seven-dimensional hand state to estimate current thumb and index-finger contact forces; empty milk containers are held out for OOD evaluation. EgoForce combines visual–state encoding, a four-frame GRU, Laplace distribution heads, and two softly weighted residual experts, and is compared with eight baselines. Separate experiments perturb force commands for a simulation policy and track prescribed forces with a physical gripper to illustrate the application context. The core contribution is a controlled data resource and an offline regression benchmark, rather than a demonstrated pre-contact physical-reasoning or closed-loop manipulation system.

Critical strengths
  1. The dataset design has clear research value. FEEL covers fragile, deformable, and variable-load properties, including fracture boundaries, visible liquid levels, and opaque packaging. This is more informative for studying when visual cues fail than simply aggregating ordinary grasps. The hardware inventory, event alignment, spatial calibration, and data-management tools provide concrete reproduction guidance. These strengths concern the design and procedures documented in the paper, not an independent verification of externally released resources.
  2. The evaluation distinguishes force magnitude, temporal behavior, and contact, and reports unfavorable results. Table 1 separately reports contact MAE, R², episode-level correlation, and contact F1 over all frames, while also comparing different action chunk lengths. EgoForce achieves a test MAE of 7.3386 N versus 7.6107 N for the best external-baseline configuration, GR00T. It further shows substantial OOD improvements in point estimates from the expert branch, without hiding the F1 decrease or the advantages of the pooling variant.
Critical weaknesses
  1. The central concept is not sufficiently operationalized. The introduction and Figure 1 emphasize pre-contact load anticipation, overshoot caused by misleading visual cues, and subsequent correction. However, the supervised target is current human-applied force, with observations that include post-grasp images and hand motion, allowing models to exploit post-contact deformation and action cues (Sections 3.3 and 4.1). The material-level results and three trajectory examples do not isolate load counterfactuals with matched appearance, pre-contact prediction, or fracture-boundary judgments, and therefore do not distinguish physical anticipation from correlational regression (Figures 5–6). The simulation perturbs Tabero-VTLA, while the physical gripper tracks manually prescribed targets rather than EgoForce predictions. The paper acknowledges that transfer remains open; these experiments should delimit the contribution rather than serve as evidence of learned embodied control.
  2. The robustness and source of the architectural advantage remain unclear. The main comparison uses one seed and fixed training steps, with different batch sizes, trainable parameter counts, and adaptation strategies. It does provide training curves, but decreasing training loss does not replace validation-based tuning or uncertainty estimates from repeated training (Tables 3–4, Figure 13). On the test set, the expert branch reduces MAE only from 7.3905 to 7.3386 N, while F1 decreases from 0.8600 to 0.8395. Four-frame pooling achieves better test MAE and R², so the GRU is not uniformly beneficial. The experts' OOD gains are noteworthy, but the full model's R² remains −0.7672; higher correlation cannot substitute for accurate force magnitude (Table 5). Moreover, the ablations do not isolate the Laplace objective, auxiliary contact supervision, or parameter-matched residual heads. The distinctive contribution of the claimed physical inductive bias over generic regression design is therefore unresolved.
  3. Split independence and measurement controls require further documentation. The paper specifies episode-level separation and four source sessions in OOD, but partitions some data randomly by episode and the remainder by session. Without a category–demonstrator–session manifest, it is difficult to assess the contribution of shared backgrounds and individual force habits. Cross-person testing and broader OOD evaluation are absent. The paper describes synchronization procedures without dataset-level alignment residuals, force-label zero-offset treatment, or calibration-error statistics. It also does not specify which data are used to fit quantile normalization. Contact is defined as strictly greater than 0 N, but contact prevalence, an always-positive baseline, and threshold sensitivity are not reported, limiting interpretation of F1. These are unresolved confounds and information gaps, not demonstrated leakage or labeling errors.
Questions for authors
  1. Could you use the existing liquid-level, misleading-load, and fracture examples to report results separately for pre-contact, initial-contact, and stable-grasp phases, and compare against hand-state-only or simple category-prior baselines to test the mechanisms described in Figure 1?
  2. Are the rankings in Tables 1 and 5 stable across repeated seeds, explicit validation-based selection rules, and matched observation histories? In particular, what are the confidence intervals for the expert branch's small in-distribution improvement?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 1/6

Thank you for recognizing FEEL's controlled physical conditions, continuous fingertip-force supervision, and multidimensional evaluation. To address your concerns, we report pre-contact and phase-specific evaluations, three-seed comparisons with all baselines, targeted component ablations, and detailed data splits and contact-score analyses for the current dataset. Learned-model results on the fixed Train/ID/OOD split are reported as mean ±\pm sample standard deviation across three random seeds. We additionally report 95% confidence intervals for the paired improvements from the expert branch. The independent cross-demonstrator evaluation is described separately in Section 5 of this response.

  1. Response to W1 and Q1: Pre-contact prediction, initial-contact evaluation, and stable-grasping evaluation

We evaluate three stages separately: before contact, initial contact, and stable grasping. At each stage, we compare full RGB-plus-hand-state input, hand-state-only input, and the training-set per-finger mean-force reference.

1.1. Strict pre-contact prediction

Let t0t_0 be the first frame in an episode at which the ground-truth force of either finger exceeds 0 N. The model receives four consecutive actual observation frames from t0−4t_0 - 4 through t0−1t_0 - 1 and uses the output for this window at t0−1t_0 - 1 to predict the two-channel fingertip force at t0t_0. All inputs precede first contact. At 10 Hz, the final input and the target are approximately 0.1 s apart.

This evaluation directly uses the existing step-10,000 checkpoints and their corresponding normalization parameters. The full model and the hand-state-only model use observation histories of the same length. The training-set mean-force reference uses the per-finger means from the training data. Evaluation covers 302 ID episodes and 113 Empty Milk OOD episodes. MAE is first averaged over the two force channels and then averaged equally across episodes.

MethodID first-contact-frame MAE ↓\downarrow (N)OOD first-contact-frame MAE ↓\downarrow (N)
EgoForce (full)3.662±0.341\mathbf{3.662 \pm 0.341}1.349±0.254\mathbf{1.349 \pm 0.254}
Hand state only3.976±2.1633.976 \pm 2.1635.739±2.9725.739 \pm 2.972
Training-set per-finger mean force13.60914.567

On Empty Milk OOD, the full model achieves a mean MAE of 1.349 N, a 76.5% reduction from 5.739 N for hand-state-only input; the training-set mean-force reference yields 14.567 N. On ID, the full model achieves 3.662 N, compared with 3.976 N for hand-state-only input and 13.609 N for the training-set mean-force reference. This restriction on input timing directly tests the ability of pre-contact observations to predict first-contact force and quantifies the contribution of visual information under the hidden-load condition.

1.2. Initial contact and stable grasping

Initial contact is defined as the three sampled frames starting at t0t_0, namely t0t_0, t0+1t_0 + 1, and t0+2t_0 + 2, corresponding to approximately [0,0.3)[0, 0.3) s after contact. Stable grasping uses the existing force-plateau label, phase=2\mathrm{phase} = 2, requires at least one finger to remain in contact, and excludes the initial-contact window. The MAE of the two fingertip-force channels is computed separately for each phase: we first average over that phase's frames and force channels within each episode, and then average equally across episodes.

Initial contact and stable grasping cover 871 and 787 ID episodes, respectively, and 261 and 253 OOD episodes, respectively.

SplitPhaseEgoForce MAE ↓\downarrow (N)Hand-state-only MAE ↓\downarrow (N)Training-set mean-force MAE ↓\downarrow (N)
IDInitial contact6.303±0.384\mathbf{6.303 \pm 0.384}6.726±1.2316.726 \pm 1.23111.980
IDStable grasping7.698±0.301\mathbf{7.698 \pm 0.301}10.856±0.55210.856 \pm 0.55210.233
OODInitial contact2.935±0.395\mathbf{2.935 \pm 0.395}7.575±3.6997.575 \pm 3.69912.481
OODStable grasping3.622±0.577\mathbf{3.622 \pm 0.577}7.069±3.1767.069 \pm 3.1769.837

During initial contact, the full model reduces mean MAE relative to hand-state-only input by 6.3% on ID and 61.3% on Empty Milk OOD. During stable grasping, the respective reductions are 29.1% and 48.8%. The full model also has lower mean error than the training-set mean-force reference in both phases on both sets, demonstrating the predictive value of visual information during early loading and sustained contact.

AuthorsOpenReview ↗

Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 2/6

Continued: Section 1 (W1 and Q1)

1.3. Specific conditions involving liquid levels, hidden loads, and fragile objects

For the object conditions you identified, the following table reports all three stages for two liquid levels, Empty Bottle and Full Bottle, the hidden-load Empty Milk condition, and the fragile Chip object. Each row compares the two models on the same episodes under the same condition and at the same stage. Pre-contact rows evaluate force at the first contact frame; initial-contact and stable-grasping rows evaluate force within the corresponding phase.

Object conditionEvaluation stageEpisodesEgoForce MAE ↓\downarrow (N)Hand-state-only MAE ↓\downarrow (N)
Empty Bottle (0% fill, ID)Pre-contact prediction272.252±0.608\mathbf{2.252 \pm 0.608}4.077±3.3704.077 \pm 3.370
Empty Bottle (0% fill, ID)Initial contact282.048±0.376\mathbf{2.048 \pm 0.376}3.340±2.6493.340 \pm 2.649
Empty Bottle (0% fill, ID)Stable grasping281.840±0.213\mathbf{1.840 \pm 0.213}4.725±3.6904.725 \pm 3.690
Full Bottle (100% fill, ID)Pre-contact prediction483.292±1.058\mathbf{3.292 \pm 1.058}3.693±2.5003.693 \pm 2.500
Full Bottle (100% fill, ID)Initial contact1374.750±0.463\mathbf{4.750 \pm 0.463}5.434±0.5505.434 \pm 0.550
Full Bottle (100% fill, ID)Stable grasping1288.065±0.231\mathbf{8.065 \pm 0.231}10.625±0.67510.625 \pm 0.675
Chip (fragile object, ID)Pre-contact prediction250.817±0.126\mathbf{0.817 \pm 0.126}2.312±1.9642.312 \pm 1.964
Chip (fragile object, ID)Initial contact760.969±0.158\mathbf{0.969 \pm 0.158}2.085±1.4822.085 \pm 1.482
Chip (fragile object, ID)Stable grasping761.043±0.241\mathbf{1.043 \pm 0.241}2.900±2.9152.900 \pm 2.915
Empty Milk (hidden load, OOD)Pre-contact prediction1131.349±0.254\mathbf{1.349 \pm 0.254}5.739±2.9725.739 \pm 2.972
Empty Milk (hidden load, OOD)Initial contact2612.935±0.395\mathbf{2.935 \pm 0.395}7.575±3.6997.575 \pm 3.699
Empty Milk (hidden load, OOD)Stable grasping2533.622±0.577\mathbf{3.622 \pm 0.577}7.069±3.1767.069 \pm 3.176

For both Empty Bottle and Full Bottle, the full model has lower mean MAE than the hand-state-only model at all three stages. For example, on Full Bottle, first-contact force-prediction error decreases from 3.693 N to 3.292 N, initial-contact error from 5.434 N to 4.750 N, and stable-grasping error from 10.625 N to 8.065 N.

For Chip, mean MAE is 0.817, 0.969, and 1.043 N for pre-contact prediction, initial contact, and stable grasping, respectively, compared with 2.312, 2.085, and 2.900 N for hand-state-only input. These results quantify early and sustained force estimation during interactions with a fragile object. The three Empty Milk stages evaluate pre-contact anticipation, initial loading, and sustained-contact estimation under a hidden load.

Through these evaluations with explicit temporal conditions, FEEL separately quantifies pre-contact force anticipation and force estimation during contact. The input comparisons demonstrate the contribution of RGB to these capabilities.

In the Introduction, Figure 1, Section 5.1, and Conclusion, we will ground the physical-intuition conclusions in pre-contact prediction of first-contact force and continuous force estimation during initial loading and stable grasping. We will specify the evaluation scope through the liquid-level variations, the hidden-load Empty Milk condition, and interactions with the fragile Chip object. Overshoot and correction, fracture-boundary assessment, and closed-loop robot control driven by predicted force will be presented as directions for future work.

  1. Response to W2 and Q2: Repeated training, baseline rankings, and checkpoint selection

We trained EgoForce and all eight external baselines with seeds 1000, 2000, and 3000. Every run used a budget of 10,000 steps, and we consistently evaluated the checkpoint saved at step 10,000. The complete model comparisons are shown below.

ID: Three-seed results for all baselines

ModelContact MAE ↓\downarrow (N)Contact R2R^2 ↑\uparrowContact Pearson rr ↑\uparrowContact F1F_1 ↑\uparrow
EgoForce (full)7.546±0.240\mathbf{7.546 \pm 0.240}0.156±0.041\mathbf{0.156 \pm 0.041}0.277±0.020\mathbf{0.277 \pm 0.020}0.842±0.023\mathbf{0.842 \pm 0.023}
ACT8.346±0.1368.346 \pm 0.1360.053±0.0150.053 \pm 0.0150.200±0.0110.200 \pm 0.0110.801±0.0100.801 \pm 0.010
OpenTouch7.681±0.2257.681 \pm 0.2250.102±0.0500.102 \pm 0.0500.203±0.0200.203 \pm 0.0200.821±0.0070.821 \pm 0.007
π0\pi_09.460±0.1889.460 \pm 0.188−0.129±0.044-0.129 \pm 0.0440.098±0.0190.098 \pm 0.0190.805±0.0230.805 \pm 0.023
π0.5\pi_{0.5}9.189±0.1699.189 \pm 0.169−0.131±0.040-0.131 \pm 0.0400.120±0.0080.120 \pm 0.0080.826±0.0100.826 \pm 0.010
SmolVLA9.367±0.0589.367 \pm 0.058−0.162±0.011-0.162 \pm 0.0110.087±0.0060.087 \pm 0.0060.781±0.0070.781 \pm 0.007
XVLA12.236±1.14912.236 \pm 1.149−0.683±0.197-0.683 \pm 0.1970.011±0.0070.011 \pm 0.0070.817±0.0180.817 \pm 0.018
WALL-X7.865±0.0357.865 \pm 0.0350.098±0.0170.098 \pm 0.0170.099±0.0050.099 \pm 0.0050.830±0.0070.830 \pm 0.007
GR00T7.551±0.0457.551 \pm 0.0450.057±0.0260.057 \pm 0.0260.222±0.0160.222 \pm 0.0160.835±0.0040.835 \pm 0.004
AuthorsOpenReview ↗

Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 3/6

Continued: Section 2 (W2 and Q2)

OOD: Three-seed results for all baselines

ModelContact MAE ↓\downarrow (N)Contact R2R^2 ↑\uparrowContact Pearson rr ↑\uparrowContact F1F_1 ↑\uparrow
EgoForce (full)3.357±0.441\mathbf{3.357 \pm 0.441}−0.042±0.372\mathbf{-0.042 \pm 0.372}0.353±0.045\mathbf{0.353 \pm 0.045}0.865±0.019\mathbf{0.865 \pm 0.019}
ACT4.510±0.9814.510 \pm 0.981−1.626±2.178-1.626 \pm 2.1780.335±0.0190.335 \pm 0.0190.815±0.0520.815 \pm 0.052
OpenTouch9.790±4.4749.790 \pm 4.474−8.374±8.022-8.374 \pm 8.0220.331±0.0110.331 \pm 0.0110.839±0.0070.839 \pm 0.007
π0\pi_08.484±0.4888.484 \pm 0.488−5.794±0.838-5.794 \pm 0.8380.208±0.0090.208 \pm 0.0090.814±0.0160.814 \pm 0.016
π0.5\pi_{0.5}6.821±0.3786.821 \pm 0.378−3.589±0.429-3.589 \pm 0.4290.169±0.0160.169 \pm 0.0160.843±0.0000.843 \pm 0.000
SmolVLA16.595±0.40116.595 \pm 0.401−22.785±0.575-22.785 \pm 0.5750.134±0.0050.134 \pm 0.0050.832±0.0030.832 \pm 0.003
XVLA12.133±3.00512.133 \pm 3.005−11.190±4.266-11.190 \pm 4.2660.040±0.0120.040 \pm 0.0120.835±0.0080.835 \pm 0.008
WALL-X10.562±2.06010.562 \pm 2.060−6.510±2.757-6.510 \pm 2.7570.249±0.0110.249 \pm 0.0110.844±0.0010.844 \pm 0.001
GR00T5.146±0.3315.146 \pm 0.331−1.025±0.319-1.025 \pm 0.3190.344±0.0190.344 \pm 0.0190.852±0.0030.852 \pm 0.003

Across three-seed averages, EgoForce achieves the best mean on all four metrics in both ID and OOD among the methods in this external-baseline comparison. Its ID contact MAE is 7.546 N, compared with 7.551 N for GR00T. EgoForce's R2R^2, Pearson rr, and F1F_1 are 0.156, 0.277, and 0.842, respectively.

On OOD, EgoForce achieves a mean contact MAE of 3.357 N, a reduction of 25.6% relative to ACT (4.510 N), the external baseline with the lowest error, and 34.8% relative to GR00T (5.146 N). Its mean OOD Pearson rr is 0.353 and F1F_1 is 0.865. These comparisons jointly report force magnitude, correlation, and contact scores, while explicitly showing the variation across seeds for each result.

We also checked the rankings separately for each seed. Among EgoForce and the eight external baselines, the full model ranks in the top three for contact MAE on ID and in the top two on Empty Milk OOD for all three seeds. Its within-episode Pearson rr on ID ranks first for every seed. The preceding tables provide the aggregate comparisons based on three-seed means. Every configuration is evaluated using its fixed step-10,000 checkpoint, with the evaluation random seed fixed at 1000. The same selection rule applies to all models.

Training settings and model capacity. We follow the training hyperparameters and optimization settings recommended by each baseline's original method. All comparisons use the ForceV7 Train/ID/OOD split, 10,000 optimizer updates, single-step two-channel force output, and the same evaluation protocol to make the comparison as fair as possible. The table reports effective batch size per update and trainable parameter counts; M denotes millions of parameters.

ModelEffective batch per updateTrainable parameters (M)
EgoForce (full)436.177
ACT434.172
OpenTouch320.756
π0\pi_08578.037
π0.5\pi_{0.5}32693.422
SmolVLA899.881
XVLA4879.482
WALL-X44224.039
GR00T321068.806

EgoForce uses a four-frame history, while the eight external baselines use single-frame input. Section 3 compares four-frame GRU, four-frame pooling, and single-frame EgoForce to examine aggregation method and history length. Section 4 evaluates the expert architecture through a parameter-matched residual-head comparison with the same backbone, batch size, and four-frame history.

  1. Response to W2 and Q2: Expert branch, GRU, observation history, and confidence intervals for improvements

3.1. Expert branch, GRU, and observation history

The full model, the model without the expert branch, and the four-frame pooling model all use four observations. The GRU/pooling comparison fixes the observation length and compares aggregation methods. The single-frame model uses the current RGB image and hand state to quantify the benefit of observation history. Every configuration below uses three seeds and 10,000 training steps per seed.

ID: Three-seed results for the relevant ablations

ModelContact MAE ↓\downarrow (N)Contact R2R^2 ↑\uparrowContact Pearson rr ↑\uparrowContact F1F_1 ↑\uparrow
EgoForce (full)7.546±0.240\mathbf{7.546 \pm 0.240}0.156±0.041\mathbf{0.156 \pm 0.041}0.277±0.020\mathbf{0.277 \pm 0.020}0.842±0.023\mathbf{0.842 \pm 0.023}
Without expert branch7.601±0.2857.601 \pm 0.2850.141±0.0520.141 \pm 0.0520.269±0.0170.269 \pm 0.0170.838±0.0190.838 \pm 0.019
Four-frame pooling / without GRU7.841±0.4337.841 \pm 0.4330.101±0.0710.101 \pm 0.0710.235±0.0280.235 \pm 0.0280.798±0.0420.798 \pm 0.042
Single-frame RGB + hand state7.929±0.4717.929 \pm 0.4710.095±0.0570.095 \pm 0.0570.238±0.0200.238 \pm 0.0200.808±0.0370.808 \pm 0.037
Reviewer_XwAqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FEEL, a controlled dataset of 3,069 egocentric RGB–touch–pose episodes from four demonstrators across ten object and fill-state categories. Models predict current thumb and index-finger forces from RGB and a 7-D hand state; tactile readings supply supervision. The benchmark includes fragile, deformable, variable-load, and visually ambiguous interactions, with empty milk containers reserved for out-of-distribution evaluation. The proposed 36.18 M-parameter EgoForce combines visual–state encoding, four-frame GRU aggregation, shared Laplace force heads, and two softly weighted residual experts. Against eight adapted baselines, it obtains test contact MAE of 7.3386 N and contact R² of 0.1965. Additional experiments perturb force commands in simulation and track prescribed forces on a tactile gripper. These robot studies establish the relevance and feasibility of force control, but do not deploy EgoForce predictions.

Critical strengths
  1. A useful controlled data resource for continuous fingertip-force estimation. Section 3 combines synchronized egocentric vision, hand tracking, and tactile measurements with deliberate variation in fragility, deformation, visible fill level, and opaque-container load. Appendix A provides hardware and synchronization details as well as a collection and alignment interface. Continuous force targets complement contact-classification benchmarks, and the proposed release of data, code, and hardware blueprints could enable replication and extensions. This is the strongest contribution because it makes a physically meaningful target available under a concrete collection protocol.

  2. The benchmark exposes useful differences between error magnitude, temporal tracking, and contact detection. Tables 1 and 5 report several complementary metrics rather than summarizing performance with one score. EgoForce improves test contact MAE over GR00T, 7.3386 versus 7.6107 N, while its shared-head variant has higher contact F1. Appendix C also shows that recurrent aggregation and residual experts affect test and OOD metrics differently. The comparisons across ACT, adapted VLAs, and frozen visual features are informative within this protocol, and the disclosure of negative OOD R² helps readers distinguish temporal correlation from reliable force calibration.

Critical weaknesses

The main weakness is the gap between the paper’s physical-intuition framing and its experimental validation. Measured human fingertip forces provide meaningful supervision, and early force dynamics could serve as behavioral proxies for physical expectations. However, the evaluation aggregates current-force prediction across interactions without separately analyzing initial loading, expectation-induced overshoot, or subsequent correction under visual–physical mismatch. Consequently, the reported scores do not establish whether models capture the anticipatory physical priors highlighted in the introduction or primarily estimate ongoing contact forces from deformation and hand motion. The potential application to adaptive manipulation is compelling, but the supporting robot experiments do not evaluate EgoForce: the simulation perturbs force commands from a separate policy (§5.3), while the real gripper tracks prescribed force targets using tactile feedback (§5.4). Neither experiment demonstrates that the learned estimates improve action adjustment, manipulation success, or protection of fragile objects, and transfer from human demonstrations to robot contact conditions remains untested. These limitations are consequential because they leave both the interpretation of FEEL as a diagnostic benchmark for physical intuition and its proposed connection to adaptive manipulation insufficiently substantiated; the evidence currently supports the narrower contribution of offline human fingertip-force estimation.

Questions for authors
  1. Clarification—validity of the physical-intuition probes. In the visual–physical mismatch trials, were demonstrators unaware of the actual load, and how were trial order and repeated exposure controlled? What evidence in the collected data establishes that these conditions induced an incorrect initial expectation followed by correction? This would help determine whether the benchmark captures the anticipatory behavior motivating the paper.

  2. Clarification—interpretation of benchmark scores. What justifies interpreting improvements in aggregate current-force prediction as stronger physical intuition? Does the existing evaluation distinguish performance during initial loading from sustained contact, or otherwise establish sensitivity to expectation-induced force dynamics?

  3. Clarification—support for adaptive manipulation. Since the simulation uses a separate policy and the real-gripper experiment tracks prescribed targets, what conclusions about the utility of EgoForce for adaptive manipulation are supported by the present results? Please distinguish demonstrated findings from the proposed future integration into robot control, including any existing evidence concerning transfer from human to robot contact conditions.

If these questions are addressed properly, I am willing to keep my rating at 3.

5 comments
AuthorsOpenReview ↗

Response to Reviewer XwAq: Phase-Specific Force Estimation, Hidden-Load Conditions, and Relevance to Robotic Manipulation - Part 1/3

Thank you for recognizing the value of FEEL's continuous fingertip-force data and estimation benchmark, and for your specific suggestions on the interpretation of physical intuition and the evidence for robotic applications. Below, we address the main concerns raised in your Weakness and Questions, identify the corresponding questions in each section, and present the additional evaluation results and planned manuscript revisions.

Following the other two reviewers' suggestions for experiments with multiple random seeds, we conducted additional runs with seeds 1000, 2000, and 3000. The original experiments used NVIDIA A100 GPUs; changes in available computing resources have made A100 GPUs unavailable. To ensure fair comparisons under the same hardware conditions, we followed the original experimental protocol and retrained and tested EgoForce and all baselines on NVIDIA H20 GPUs. All learned-model evaluation results below report the mean ± standard deviation across these three random seeds.

  1. Phase-specific force estimation and input diagnostics (W1: evaluation concerns; Q2)

To address the concern that aggregate metrics do not distinguish interaction phases, we added evaluations across three seeds for initial loading and stable contact, comparing full RGB-plus-hand-state input, hand state only, and the training-set per-finger mean-force reference. These evaluations quantify estimation accuracy during contact establishment and sustained contact separately, and test the contribution of visual information beyond hand state. The task defined in Sections 3.3 and 4.1 estimates current two-finger contact forces from observed egocentric RGB and hand state.

1.1. Phase definitions, comparison settings, and computation

Let t0t_0 denote the first frame in an episode whose ground-truth force label exceeds 0 N for either finger. The initial-contact phase comprises the three consecutive sampled frames starting at t0t_0, namely t0t_0, t0+1t_0+1, and t0+2t_0+2, corresponding to approximately [0,0.3)[0,0.3) s after contact at a sampling rate of 10 Hz. Stable grasping uses the dataset's existing force-plateau label phase=2\mathrm{phase}=2, requires at least one finger to remain in contact, and excludes the initial-contact window. Initial loading and the subsequent force plateau are therefore evaluated separately.

The full model receives four frames of RGB and hand state. The hand-state-only model sets the RGB input to zero throughout both training and evaluation, while retaining the same four-frame hand-state history. Each model is independently trained for 10,000 steps with three random seeds. Evaluation uses the archived predictions from the corresponding checkpoints and the original training normalization parameters. The training-set mean-force reference uses two fixed outputs computed from the training data.

We regroup the existing model predictions by these phases and compute the MAE of the two fingertip-force channels within each phase. We first average over the frames and the two force channels for that phase within each episode, and then average equally across episodes. Initial contact and stable grasping include 871 and 787 ID episodes, respectively, and 261 and 253 OOD episodes, respectively.

1.2. Results across three seeds and their interpretation

SplitPhaseEgoForce: RGB + hand state MAE ↓ (N)Hand-state-only MAE ↓ (N)Training-set per-finger mean-force MAE ↓ (N)
IDInitial contact6.303 ± 0.3846.726 ± 1.23111.980
IDStable grasping7.698 ± 0.30110.856 ± 0.55210.233
OOD (Empty Milk)Initial contact2.935 ± 0.3957.575 ± 3.69912.481
OOD (Empty Milk)Stable grasping3.622 ± 0.5777.069 ± 3.1769.837
AuthorsOpenReview ↗

Response to Reviewer XwAq: Phase-Specific Force Estimation, Hidden-Load Conditions, and Relevance to Robotic Manipulation - Part 2/3

1.2. Results across three seeds and their interpretation (continued; W1/Q2)

During initial contact, the full model achieves a mean MAE of 6.303 N on ID, compared with 6.726 N for the hand-state-only model and 11.980 N for the training-set mean-force reference. On Empty Milk OOD, the corresponding values are 2.935 N, 7.575 N, and 12.481 N. Relative to the hand-state-only model, mean error decreases by 6.3% on ID and 61.3% on OOD. These results directly address force estimation when contact is first established and quantify the role of visual information at this early stage.

During stable grasping, the full model achieves a mean MAE of 7.698 N on ID, compared with 10.856 N for the hand-state-only model and 10.233 N for the training-set mean-force reference. On OOD, the respective values are 3.622 N, 7.069 N, and 9.837 N. Relative to the hand-state-only model, the full model reduces mean error by 29.1% on ID and 48.8% on OOD, showing that visual information also improves force-magnitude estimation during sustained contact.

This evaluation distinguishes initial loading from sustained contact and quantifies force-estimation accuracy in both phases. The hand-state-only comparison quantifies the contribution of RGB beyond hand-motion information, while the training-set mean-force reference tests how much a fixed force prior can explain. Compared with both references, the full model achieves lower mean error in both phases on ID and Empty Milk OOD, providing phase-specific evidence for continuous force estimation during initial contact and sustained grasping.

We will add these phase definitions, results across three seeds, and input comparisons to Section 5.1, and tie the corresponding conclusions to force-estimation accuracy during initial loading and sustained contact, as well as generalization across conditions.

  1. Hidden-load collection protocol and behavioral interpretation (W1: probe interpretation; Q1)

The Empty Milk OOD condition in the preceding evaluation examines force estimation when object appearance does not directly reveal the actual load.

2.1. Load knowledge and trial order

Grasping trials across different objects and load conditions were arranged in a randomized order. Before data collection began for the visual–physical mismatch conditions, the demonstrators did not know the actual object weight, and the collection staff did not provide weight information. The Empty Milk condition uses opaque packaging that conceals the internal load.

2.2. Interpretation of an incorrect initial expectation followed by correction

The hidden-load condition is designed to study how people form load expectations and adjust their applied forces after contact when object appearance does not directly reveal the actual load.

In response to your concern about an incorrect initial expectation followed by correction, we will clarify this design objective in the Introduction and present expectation, overshoot, and correction as the research motivation in the explanation of Figure 1. Descriptions of human behavior will focus on the forces actually recorded during contact establishment and subsequent holding. The Conclusion will emphasize FEEL's synchronized vision, hand-state, and continuous fingertip-force data collected under controlled physical conditions, together with its current-force estimation benchmark.

  1. Manipulation value of force information and evidence for transfer (W1: application concerns; Q3)

Our robot experiments support two conclusions directly relevant to adaptive manipulation: force-command accuracy affects success in the evaluated task, and a real gripper can execute continuous target forces through tactile feedback. Prior work further demonstrates the practical benefits of force feedback in robot action generation and of human force–motion demonstrations for robot learning. We combine our results with these studies to explain the application value of FEEL's continuous-force data and estimation benchmark.

AuthorsOpenReview ↗

Response to Reviewer XwAq: Phase-Specific Force Estimation, Hidden-Load Conditions, and Relevance to Robotic Manipulation - Part 3/3

  1. Manipulation value of force information and evidence for transfer (W1: application concerns; Q3) — continued

3.1. Simulation: Effect of force-command error on manipulation success

In the Isaac Sim experiments in Section 5.3 and Appendix E, we use Tabero-VTLA for the Milk manipulation task. With paired initial states and sampling noise, controlled perturbations are applied only to the bilateral force-command channels of the policy output. Each perturbation level is evaluated over 150 trials. Representative results are shown below.

Injected force-command MAE (N)Manipulation success rate ↑
No added error84.7%
453.3%
628.0%

As the injected force-command error increases from 0 N to 4 N and 6 N, manipulation success decreases from 84.7% to 53.3% and 28.0%, respectively. This controlled comparison shows that force-command accuracy directly affects task completion for this policy and manipulation task, providing task-level support for studying continuous force-prediction accuracy.

3.2. Real-gripper experiment: Executing target forces through tactile feedback

The real-gripper experiment in Section 5.4 uses electronic-skin feedback to track preset target forces in the 1–5 N range. The MAE between sensor feedback readings and target values across the nine target-force intervals is shown below.

Evaluation windowTarget-force tracking MAE ↓ (N)
All samples in the nine target-force intervals0.0487
Final 10 s of each target-force interval0.0207

The full-interval result captures overall performance during target changes and sustained tracking. The final-10-s windows capture tracking performance toward the end of each target interval. These results demonstrate the real gripper's ability to execute continuous target forces through tactile feedback, providing hardware evidence for using force as a manipulation control variable.

3.3. Related work: Manipulation value of force feedback and transfer of human force demonstrations

ForceVLA (Yu et al., 2025) fuses real-time six-dimensional force/torque feedback with vision-language representations through a force-aware mixture-of-experts module for robot action generation. Across its five contact-rich manipulation tasks, ForceVLA achieves a mean success rate of 60.5%, compared with 37.3% for the π0\pi_0-base baseline without force feedback, a gain of 23.2 percentage points. This result shows that force feedback with an appropriate fusion design can improve contact-rich manipulation, providing empirical evidence for incorporating force information into robot decision-making.

ForceMimic (Liu et al., 2025) uses the handheld ForceCapture tool to record motion and interaction forces during human demonstrations. It converts these recordings into representations for robot learning, trains HybridIL to predict force and pose, and executes robotic peeling through hybrid force–position control. Under the success criterion of a peel length exceeding 10 cm, it achieves an 85% success rate, compared with 55% for the vision-and-motion baseline Raw DP. This work demonstrates a concrete path from human-collected force–motion demonstrations to robot skill learning, providing a directly relevant precedent for studying the transfer of force knowledge.

FEEL provides synchronized egocentric vision, hand state, and continuous fingertip force, together with a force-estimation benchmark, supplying data and a common evaluation basis for learning force information from human interactions. In Sections 5.3–5.4, we will specify what each experiment evaluates: the effect of force-command error on task success and continuous target-force execution through tactile feedback. ForceVLA and ForceMimic provide application precedents for using force information in robot policies and transferring human demonstrations. Integrating EgoForce predictions into robot feedback control and studying the transfer of human force knowledge to robot contact conditions are future integration directions.

References

Yu, Jiawen, Hairuo Liu, Qiaojun Yu, Jieji Ren, Ce Hao, Haitong Ding, Guangyu Huang, Guofan Huang, Yan Song, Panpan Cai, Wenqiang Zhang, and Cewu Lu. ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation. Advances in Neural Information Processing Systems, 38, 93409–93439, 2025.

Liu, Wenhai, Junbo Wang, Yiming Wang, Weiming Wang, and Cewu Lu. ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation. IEEE International Conference on Robotics and Automation (ICRA), 2025.

Reviewer_vif7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FEEL, an egocentric benchmark for estimating the current thumb and index-finger contact forces from RGB observations and a 7-D hand state. It contains 3,069 synchronized vision-touch-pose episodes from four demonstrators across ten object/fill-state categories, organized around fragile, deformable, and variable-load properties. Nine categories are used for training and in-distribution testing, while empty milk containers form the held-out OOD condition. The authors also propose EgoForce, an ACT-based model combining four-frame GRU aggregation, shared Laplace force heads, and two softly weighted residual experts. They compare it with eight adapted baselines, including regression policies, VLA models, and an OpenTouch-derived visual regression baseline. EgoForce reports test contact MAE of 7.3386 N and contact R² of 0.1965; on the held-out empty-container condition, it reports MAE of 4.3963 N, R² of -0.7672, and episode-averaged contact correlation of 0.4067. Separate simulation and real-gripper experiments examine the effects of force-command errors and the feasibility of tactile-feedback force tracking.

Critical strengths
  1. A useful controlled resource for an underexplored quantitative task. FEEL combines synchronized egocentric video, hand state, and fingertip-force supervision with systematic fill-level variation and fragile/deformable interactions. Its main value is the ability to study continuous force magnitude and temporal variation rather than only semantic task completion or contact classification. The hardware blueprint, alignment UI, and planned release of the data and evaluation pipeline could make the benchmark reproducible and useful beyond this paper.

  2. Broad comparisons with clearly defined, metric-specific evaluation. The evaluation covers eight adapted baselines and reports contact MAE, contact R², within-episode correlation, and contact detection rather than relying on a single favorable score. The component ablations also expose trade-offs instead of showing uniform gains: replacing the GRU with four-frame pooling improves test MAE and R², while retaining the GRU improves all four OOD metrics; the residual experts improve OOD magnitude estimation but reduce test F1.

Critical weaknesses
  1. The physical-intuition framing is not fully matched by the reported evaluation. The introduction and Figure 1 emphasize anticipatory load estimation and force overshoot under deceptive appearance, whereas the implemented task estimates current contact force from observations that may already contain hand-motion and deformation cues (Sections 3.3 and 4.1). The held-out empty-container condition is useful, but the reported results do not separately evaluate matched visual-physical counterfactuals, pre-contact anticipation, or model overshoot and correction. Likewise, the simulation perturbations and prescribed-target gripper experiments do not connect EgoForce predictions to closed-loop control (Sections 5.3-5.4).

  2. The generality and statistical reliability of the architectural findings remain insufficiently characterized. The mixed episode/session split needs a clearer account of demonstrator, source-recording, and object-instance overlap (Section 5). The model comparisons use one training seed, and the common step budget does not isolate model scale from differences in batch size, objectives, and trainable parameters (Tables 3-4). In Table 5, residual experts improve test MAE by 0.0519 N while reducing test F1; four-frame pooling gives better test MAE and R². The hand-state-free variant also has lower OOD MAE than the full model, although its other OOD metrics are worse. The strongest expert-branch magnitude gain occurs on the sole held-out Empty Milk category. OOD contact R² remains negative despite improved within-episode correlation.

  3. Benchmark validity needs clearer label characterization and more diagnostic reference baselines. Appendix A describes per-axis tactile capture and temporal/spatial alignment, but does not clearly specify how these readings become the two scalar force targets or quantify their zeroing, measurement uncertainty, and residual synchronization error. These details matter for both errors reported in N and contact scoring at a 0 N threshold. The evaluation also lacks simple training-derived constant-force references and an always-contact reference.

Questions for authors
  1. Task scope: Which reported results directly evaluate the boundary and visual-physical mismatch probes, rather than aggregate current-force estimation? Please distinguish the operational task from the pre-contact anticipation and overshoot/correction behavior emphasized in the introduction and Figure 1, and clarify whether the framing would be narrowed where these constructs are not separately evaluated.

  2. Force labels and contact scoring: For FEEL, please specify whether each scalar target is a normal-force component, a vector magnitude, or another aggregation, and describe force calibration, zeroing, noise handling, and the mapping from the 50 Hz tactile stream to the 10 Hz RGB timeline, including residual alignment error. Please report contact prevalence and assess F1 sensitivity to noise-informed nonzero thresholds and an always-contact reference. Is the auxiliary any-finger contact head used at inference, given that reported F1 is computed per finger from predicted forces? Separately, Appendix D.1 states that the real-gripper run lacks independent absolute-force calibration; please distinguish sensor-readout tracking accuracy from independently verified absolute-force accuracy.

  3. Diagnostic baselines: Could you report state-only performance to isolate the contribution of RGB beyond hand geometry and motion, alongside the existing hand-state-free ablation? Simple per-finger constant mean/median predictors fitted on training data would also help contextualize absolute-force performance. If temporal modeling is a central claim, a single-frame comparison would distinguish the benefit of multiple observations from the choice of GRU versus four-frame pooling.

5 comments
AuthorsOpenReview ↗

Response to Reviewer vif7: Multi-Seed Comparisons, Model Ablations, and Pre-Contact Force Prediction - Part 1/4

Thank you for recognizing the design of FEEL, its evaluation of continuous fingertip force, and the baseline comparisons. Below, we report the additional experimental results addressing your specific concerns. All learned-model results on the fixed split are reported as mean ± standard deviation across three random seeds.

  1. Response to W2: Single-seed comparisons and result stability

We trained EgoForce and all eight external baselines with three random seeds each. Every run used a budget of 10,000 training steps, and we consistently evaluated the checkpoint saved at step 10,000. The complete results are provided below.

ID: Three-seed results for all baselines

ModelContact MAE ↓ (N)Contact R² ↑Contact Pearson r ↑Contact F1 ↑
EgoForce (full)7.546 ± 0.2400.156 ± 0.0410.277 ± 0.0200.842 ± 0.023
ACT8.346 ± 0.1360.053 ± 0.0150.200 ± 0.0110.801 ± 0.010
OpenTouch7.681 ± 0.2250.102 ± 0.0500.203 ± 0.0200.821 ± 0.007
π₀9.460 ± 0.188-0.129 ± 0.0440.098 ± 0.0190.805 ± 0.023
π₀.₅9.189 ± 0.169-0.131 ± 0.0400.120 ± 0.0080.826 ± 0.010
SmolVLA9.367 ± 0.058-0.162 ± 0.0110.087 ± 0.0060.781 ± 0.007
XVLA12.236 ± 1.149-0.683 ± 0.1970.011 ± 0.0070.817 ± 0.018
WALL-X7.865 ± 0.0350.098 ± 0.0170.099 ± 0.0050.830 ± 0.007
GR00T7.551 ± 0.0450.057 ± 0.0260.222 ± 0.0160.835 ± 0.004

OOD: Three-seed results for all baselines

ModelContact MAE ↓ (N)Contact R² ↑Contact Pearson r ↑Contact F1 ↑
EgoForce (full)3.357 ± 0.441-0.042 ± 0.3720.353 ± 0.0450.865 ± 0.019
ACT4.510 ± 0.981-1.626 ± 2.1780.335 ± 0.0190.815 ± 0.052
OpenTouch9.790 ± 4.474-8.374 ± 8.0220.331 ± 0.0110.839 ± 0.007
π₀8.484 ± 0.488-5.794 ± 0.8380.208 ± 0.0090.814 ± 0.016
π₀.₅6.821 ± 0.378-3.589 ± 0.4290.169 ± 0.0160.843 ± 0.000
SmolVLA16.595 ± 0.401-22.785 ± 0.5750.134 ± 0.0050.832 ± 0.003
XVLA12.133 ± 3.005-11.190 ± 4.2660.040 ± 0.0120.835 ± 0.008
WALL-X10.562 ± 2.060-6.510 ± 2.7570.249 ± 0.0110.844 ± 0.001
GR00T5.146 ± 0.331-1.025 ± 0.3190.344 ± 0.0190.852 ± 0.003

Across the three-seed averages, EgoForce achieves the best mean on all four metrics in both ID and OOD among the methods in this external-baseline comparison. On ID, EgoForce and GR00T have similar mean MAE values of 7.546 N and 7.551 N, respectively. EgoForce achieves R², Pearson r, and F1 of 0.156, 0.277, and 0.842, compared with 0.057, 0.222, and 0.835 for GR00T.

On OOD, ACT has the lowest mean MAE among the external baselines at 4.510 N, while EgoForce achieves 3.357 N, reducing mean error by 25.6%. Relative to GR00T's 5.146 N, the reduction is 34.8%.

We also checked the rankings separately for each seed. Among EgoForce and the eight external baselines, the full model ranks in the top three for contact MAE on ID and in the top two on Empty Milk OOD for all three seeds. Its within-episode Pearson r on ID ranks first for every seed. The preceding tables provide the aggregate comparisons based on three-seed means. Every configuration is evaluated using its fixed step-10,000 checkpoint, with the evaluation random seed fixed at 1000. The same selection rule applies to all models.

We follow the training hyperparameters and optimization settings recommended by each baseline's original method and use a common budget of 10,000 optimizer updates. With the same data splits and evaluation protocol, this preserves the recommended training setup for each method while using a fixed number of training steps to make the comparison as fair as possible.

AuthorsOpenReview ↗

Response to Reviewer vif7: Multi-Seed Comparisons, Model Ablations, and Pre-Contact Force Prediction - Part 2/4

  1. Response to W2 and Q3: Contributions of the expert branch, GRU, and single-frame input

We repeated the experiments without the expert branch, with four-frame pooling, and with single-frame input to examine the expert contribution, the comparison between GRU and pooling, and the contribution of multiple observations.

ID: Three-seed results for the relevant ablations

ModelContact MAE ↓ (N)Contact R² ↑Contact Pearson r ↑Contact F1 ↑
EgoForce (full)7.546 ± 0.2400.156 ± 0.0410.277 ± 0.0200.842 ± 0.023
Without expert branch7.601 ± 0.2850.141 ± 0.0520.269 ± 0.0170.838 ± 0.019
Four-frame pooling / without GRU7.841 ± 0.4330.101 ± 0.0710.235 ± 0.0280.798 ± 0.042
Single-frame RGB + hand state7.929 ± 0.4710.095 ± 0.0570.238 ± 0.0200.808 ± 0.037

OOD: Three-seed results for the relevant ablations

ModelContact MAE ↓ (N)Contact R² ↑Contact Pearson r ↑Contact F1 ↑
EgoForce (full)3.357 ± 0.441-0.042 ± 0.3720.353 ± 0.0450.865 ± 0.019
Without expert branch4.800 ± 0.494-2.667 ± 2.8120.352 ± 0.0600.856 ± 0.010
Four-frame pooling / without GRU5.276 ± 0.920-2.359 ± 1.9130.351 ± 0.0370.811 ± 0.061
Single-frame RGB + hand state4.007 ± 1.392-1.285 ± 2.4740.300 ± 0.0080.804 ± 0.042

Expert branch. Adding the expert branch reduces mean MAE from 7.601 N to 7.546 N on ID, an improvement of 0.055 N, and from 4.800 N to 3.357 N on OOD, an improvement of 1.444 N or 30.1%. The paired OOD improvements for the three seeds are 0.901, 1.866, and 1.564 N. Mean F1 across three seeds increases from 0.838 to 0.842 on ID and from 0.856 to 0.865 on OOD. The main benefit of the expert branch is in estimating force magnitude on OOD.

GRU versus four-frame pooling. Both configurations use four frames of RGB and hand state, so this comparison evaluates the temporal aggregation method. Across the three-seed averages, GRU outperforms pooling on all four metrics in both ID and OOD. Mean MAE improves by 0.295 N and 1.919 N, corresponding to reductions of 3.8% and 36.4%, respectively.

Multiple frames versus a single frame. The single-frame model uses only the current RGB image and hand state. Relative to this model, the full model improves mean MAE by 0.383 N on ID and 0.650 N on OOD, corresponding to reductions of 4.8% and 16.2%. This comparison directly quantifies the benefit of observation history and complements the GRU/pooling comparison with a fixed four-frame history.

  1. Response to W3 and Q3: RGB contribution and the constant-force reference

We added a hand-state-only model, which retains the seven-dimensional hand state and masks the image input. Its three-seed results are shown below.

SplitContact MAE ↓ (N)Contact R² ↑Contact Pearson r ↑Contact F1 ↑
ID: Hand state only9.912 ± 0.602-0.286 ± 0.1060.186 ± 0.0180.874 ± 0.005
OOD: Hand state only7.498 ± 3.436-2.849 ± 3.2240.132 ± 0.0710.874 ± 0.005

Adding RGB reduces mean contact MAE from 9.912 N to 7.546 N on ID, a reduction of 23.9%, and from 7.498 N to 3.357 N on OOD, a reduction of 55.2%.

Visual input also improves the within-sequence correlation of contact force: Pearson r increases from 0.186 to 0.277 on ID and from 0.132 to 0.353 on OOD. These results quantify the contributions of RGB to estimating force magnitude and its temporal variation.

Following your suggestion of a constant predictor, we constructed a fixed reference using the per-finger mean force in the training set. Its fixed outputs for the two channels are approximately 17.019 N and 14.065 N. The contact MAE results are shown below.

MethodID contact MAE ↓ (N)OOD contact MAE ↓ (N)
EgoForce: RGB + hand state7.546 ± 0.2403.357 ± 0.441
Hand state only9.912 ± 0.6027.498 ± 3.436
Training-set per-finger mean force10.58310.448

The full-input model has lower mean force error than both the hand-state-only model and the training-set mean-force reference on both test sets, providing quantitative evidence for the contribution of visual information beyond hand motion and a fixed force prior.

AuthorsOpenReview ↗

Response to Reviewer vif7: Multi-Seed Comparisons, Model Ablations, and Pre-Contact Force Prediction - Part 3/4

  1. Response to W1 and Q1: Pre-contact prediction

We added a strict pre-contact prediction experiment using the existing step-10,000 checkpoints.

Let t₀ denote the first contact frame. The model receives four consecutive actual observation frames from t₀−4 through t₀−1, and the prediction target is the two-channel fingertip force at the first contact frame t₀. The model output for the four-frame window ending at t₀−1 is used to predict the force at t₀. At a sampling rate of 10 Hz, this corresponds to a prediction horizon of approximately 0.1 s. All input frames precede the first contact.

This experiment directly reuses the existing model weights and normalization parameters and evaluates 302 ID episodes and 113 OOD episodes with four frames of pre-contact history.

MethodID first-contact-frame MAE ↓ (N)OOD first-contact-frame MAE ↓ (N)
EgoForce: RGB + hand state3.662 ± 0.3411.349 ± 0.254
Hand state only3.976 ± 2.1635.739 ± 2.972
Training-set per-finger mean force13.60914.567

On Empty Milk OOD, adding RGB reduces mean MAE from 5.739 N to 1.349 N, a reduction of 76.5%. On ID, mean MAE decreases from 3.976 N to 3.662 N. This experiment directly tests whether observations preceding contact can predict the force at first contact and quantifies the contribution of visual input under this condition.

In the Introduction, Figure 1, Section 5.1, and Conclusion, we will explicitly align the relevant claims with continuous contact-force estimation and the additional pre-contact prediction of first-contact force. The OOD conclusions for Empty Milk will be scoped to this held-out condition. Overshoot and correction, fracture-boundary assessment, and closed-loop robot control driven by predicted force will be presented as directions for future work.

  1. Response to W3 and Q2: Force labels, zeroing, calibration, and tactile–RGB alignment

We record fingertip forces using the PX6AX-GEN3-DP-S2015-Elite sensor and the manufacturer's factory-calibrated force outputs. Both the XYZ components and resultant forces are expressed in N. The manufacturer specifies repeatability of <0.5% FS, where FS denotes full scale.

Before data collection, the sensor automatically zeros after power-on and filters the force signal before it is written to the raw CSV files. The subsequent conversion uses these force values directly, without repeating zero-offset subtraction or applying additional force-signal smoothing or amplitude clipping.

The scalar target for each force channel is the magnitude of the corresponding sensor's three-axis resultant force vector: Fj=sqrt(Fxj2+Fyj2+Fzj2)F_j = sqrt(Fx_j^2 + Fy_j^2 + Fz_j^2), where j∈1,2j ∈ {1, 2} indexes the two sensor channels, and FxjFx_j, FyjFy_j, and FzjFz_j are the three force components of that channel. The supervision target for each frame is [F1,F2][F_1, F_2], and the model predicts these two scalars separately.

Temporal synchronization uses manual alignment to the 10 Hz video, with an estimated alignment precision of approximately one video frame (about 100 ms). The processing pipeline first obtains the resultant-force magnitude at the tactile sampling timestamps, then uses the manually determined time offset to linearly interpolate this scalar sequence to the corresponding video frame timestamps. Each frame is thus associated with two force labels in N.

  1. Response to W2: Demonstrators, collection records, and data splits

The current experiments contain 3,069 interaction episodes and 85,177 frames. Train, ID, and OOD contain 1,937, 871, and 261 episodes, respectively. Demonstrators are denoted P1–P4: Train and ID both include P1–P4, while OOD includes P1–P3.

The split uses two levels: complete interaction episodes and collection sessions. P1's Train/ID data are split by interaction episode. P2–P4 are split by complete collection session: each of their 26 sessions belongs to exactly one split, and all episodes from the same session enter that split.

At the category level, Train and ID contain the same nine object/fill-state categories. All 261 Empty Milk episodes are held out for OOD. This OOD condition evaluates the Empty Milk category, which is unseen during training.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.