- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FEEL, an egocentric force-estimation benchmark containing 3,069 interaction episodes from four demonstrators across ten object/load conditions. Models use RGB observations and a seven-dimensional hand state to estimate current thumb and index-finger contact forces; empty milk containers are held out for OOD evaluation. EgoForce combines visual–state encoding, a four-frame GRU, Laplace distribution heads, and two softly weighted residual experts, and is compared with eight baselines. Separate experiments perturb force commands for a simulation policy and track prescribed forces with a physical gripper to illustrate the application context. The core contribution is a controlled data resource and an offline regression benchmark, rather than a demonstrated pre-contact physical-reasoning or closed-loop manipulation system.
Critical strengths
- The dataset design has clear research value. FEEL covers fragile, deformable, and variable-load properties, including fracture boundaries, visible liquid levels, and opaque packaging. This is more informative for studying when visual cues fail than simply aggregating ordinary grasps. The hardware inventory, event alignment, spatial calibration, and data-management tools provide concrete reproduction guidance. These strengths concern the design and procedures documented in the paper, not an independent verification of externally released resources.
- The evaluation distinguishes force magnitude, temporal behavior, and contact, and reports unfavorable results. Table 1 separately reports contact MAE, R², episode-level correlation, and contact F1 over all frames, while also comparing different action chunk lengths. EgoForce achieves a test MAE of 7.3386 N versus 7.6107 N for the best external-baseline configuration, GR00T. It further shows substantial OOD improvements in point estimates from the expert branch, without hiding the F1 decrease or the advantages of the pooling variant.
Critical weaknesses
- The central concept is not sufficiently operationalized. The introduction and Figure 1 emphasize pre-contact load anticipation, overshoot caused by misleading visual cues, and subsequent correction. However, the supervised target is current human-applied force, with observations that include post-grasp images and hand motion, allowing models to exploit post-contact deformation and action cues (Sections 3.3 and 4.1). The material-level results and three trajectory examples do not isolate load counterfactuals with matched appearance, pre-contact prediction, or fracture-boundary judgments, and therefore do not distinguish physical anticipation from correlational regression (Figures 5–6). The simulation perturbs Tabero-VTLA, while the physical gripper tracks manually prescribed targets rather than EgoForce predictions. The paper acknowledges that transfer remains open; these experiments should delimit the contribution rather than serve as evidence of learned embodied control.
- The robustness and source of the architectural advantage remain unclear. The main comparison uses one seed and fixed training steps, with different batch sizes, trainable parameter counts, and adaptation strategies. It does provide training curves, but decreasing training loss does not replace validation-based tuning or uncertainty estimates from repeated training (Tables 3–4, Figure 13). On the test set, the expert branch reduces MAE only from 7.3905 to 7.3386 N, while F1 decreases from 0.8600 to 0.8395. Four-frame pooling achieves better test MAE and R², so the GRU is not uniformly beneficial. The experts' OOD gains are noteworthy, but the full model's R² remains −0.7672; higher correlation cannot substitute for accurate force magnitude (Table 5). Moreover, the ablations do not isolate the Laplace objective, auxiliary contact supervision, or parameter-matched residual heads. The distinctive contribution of the claimed physical inductive bias over generic regression design is therefore unresolved.
- Split independence and measurement controls require further documentation. The paper specifies episode-level separation and four source sessions in OOD, but partitions some data randomly by episode and the remainder by session. Without a category–demonstrator–session manifest, it is difficult to assess the contribution of shared backgrounds and individual force habits. Cross-person testing and broader OOD evaluation are absent. The paper describes synchronization procedures without dataset-level alignment residuals, force-label zero-offset treatment, or calibration-error statistics. It also does not specify which data are used to fit quantile normalization. Contact is defined as strictly greater than 0 N, but contact prevalence, an always-positive baseline, and threshold sensitivity are not reported, limiting interpretation of F1. These are unresolved confounds and information gaps, not demonstrated leakage or labeling errors.
Questions for authors
- Could you use the existing liquid-level, misleading-load, and fracture examples to report results separately for pre-contact, initial-contact, and stable-grasp phases, and compare against hand-state-only or simple category-prior baselines to test the mechanisms described in Figure 1?
- Are the rankings in Tables 1 and 5 stable across repeated seeds, explicit validation-based selection rules, and matched observation histories? In particular, what are the confidence intervals for the expert branch's small in-distribution improvement?
8 comments
Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 2/6
Continued: Section 1 (W1 and Q1)
1.3. Specific conditions involving liquid levels, hidden loads, and fragile objects
For the object conditions you identified, the following table reports all three stages for two liquid levels, Empty Bottle and Full Bottle, the hidden-load Empty Milk condition, and the fragile Chip object. Each row compares the two models on the same episodes under the same condition and at the same stage. Pre-contact rows evaluate force at the first contact frame; initial-contact and stable-grasping rows evaluate force within the corresponding phase.
| Object condition | Evaluation stage | Episodes | EgoForce MAE (N) | Hand-state-only MAE (N) |
|---|---|---|---|---|
| Empty Bottle (0% fill, ID) | Pre-contact prediction | 27 | ||
| Empty Bottle (0% fill, ID) | Initial contact | 28 | ||
| Empty Bottle (0% fill, ID) | Stable grasping | 28 | ||
| Full Bottle (100% fill, ID) | Pre-contact prediction | 48 | ||
| Full Bottle (100% fill, ID) | Initial contact | 137 | ||
| Full Bottle (100% fill, ID) | Stable grasping | 128 | ||
| Chip (fragile object, ID) | Pre-contact prediction | 25 | ||
| Chip (fragile object, ID) | Initial contact | 76 | ||
| Chip (fragile object, ID) | Stable grasping | 76 | ||
| Empty Milk (hidden load, OOD) | Pre-contact prediction | 113 | ||
| Empty Milk (hidden load, OOD) | Initial contact | 261 | ||
| Empty Milk (hidden load, OOD) | Stable grasping | 253 |
For both Empty Bottle and Full Bottle, the full model has lower mean MAE than the hand-state-only model at all three stages. For example, on Full Bottle, first-contact force-prediction error decreases from 3.693 N to 3.292 N, initial-contact error from 5.434 N to 4.750 N, and stable-grasping error from 10.625 N to 8.065 N.
For Chip, mean MAE is 0.817, 0.969, and 1.043 N for pre-contact prediction, initial contact, and stable grasping, respectively, compared with 2.312, 2.085, and 2.900 N for hand-state-only input. These results quantify early and sustained force estimation during interactions with a fragile object. The three Empty Milk stages evaluate pre-contact anticipation, initial loading, and sustained-contact estimation under a hidden load.
Through these evaluations with explicit temporal conditions, FEEL separately quantifies pre-contact force anticipation and force estimation during contact. The input comparisons demonstrate the contribution of RGB to these capabilities.
In the Introduction, Figure 1, Section 5.1, and Conclusion, we will ground the physical-intuition conclusions in pre-contact prediction of first-contact force and continuous force estimation during initial loading and stable grasping. We will specify the evaluation scope through the liquid-level variations, the hidden-load Empty Milk condition, and interactions with the fragile Chip object. Overshoot and correction, fracture-boundary assessment, and closed-loop robot control driven by predicted force will be presented as directions for future work.
- Response to W2 and Q2: Repeated training, baseline rankings, and checkpoint selection
We trained EgoForce and all eight external baselines with seeds 1000, 2000, and 3000. Every run used a budget of 10,000 steps, and we consistently evaluated the checkpoint saved at step 10,000. The complete model comparisons are shown below.
ID: Three-seed results for all baselines
| Model | Contact MAE (N) | Contact | Contact Pearson | Contact |
|---|---|---|---|---|
| EgoForce (full) | ||||
| ACT | ||||
| OpenTouch | ||||
| SmolVLA | ||||
| XVLA | ||||
| WALL-X | ||||
| GR00T |
Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 3/6
Continued: Section 2 (W2 and Q2)
OOD: Three-seed results for all baselines
| Model | Contact MAE (N) | Contact | Contact Pearson | Contact |
|---|---|---|---|---|
| EgoForce (full) | ||||
| ACT | ||||
| OpenTouch | ||||
| SmolVLA | ||||
| XVLA | ||||
| WALL-X | ||||
| GR00T |
Across three-seed averages, EgoForce achieves the best mean on all four metrics in both ID and OOD among the methods in this external-baseline comparison. Its ID contact MAE is 7.546 N, compared with 7.551 N for GR00T. EgoForce's , Pearson , and are 0.156, 0.277, and 0.842, respectively.
On OOD, EgoForce achieves a mean contact MAE of 3.357 N, a reduction of 25.6% relative to ACT (4.510 N), the external baseline with the lowest error, and 34.8% relative to GR00T (5.146 N). Its mean OOD Pearson is 0.353 and is 0.865. These comparisons jointly report force magnitude, correlation, and contact scores, while explicitly showing the variation across seeds for each result.
We also checked the rankings separately for each seed. Among EgoForce and the eight external baselines, the full model ranks in the top three for contact MAE on ID and in the top two on Empty Milk OOD for all three seeds. Its within-episode Pearson on ID ranks first for every seed. The preceding tables provide the aggregate comparisons based on three-seed means. Every configuration is evaluated using its fixed step-10,000 checkpoint, with the evaluation random seed fixed at 1000. The same selection rule applies to all models.
Training settings and model capacity. We follow the training hyperparameters and optimization settings recommended by each baseline's original method. All comparisons use the ForceV7 Train/ID/OOD split, 10,000 optimizer updates, single-step two-channel force output, and the same evaluation protocol to make the comparison as fair as possible. The table reports effective batch size per update and trainable parameter counts; M denotes millions of parameters.
| Model | Effective batch per update | Trainable parameters (M) |
|---|---|---|
| EgoForce (full) | 4 | 36.177 |
| ACT | 4 | 34.172 |
| OpenTouch | 32 | 0.756 |
| 8 | 578.037 | |
| 32 | 693.422 | |
| SmolVLA | 8 | 99.881 |
| XVLA | 4 | 879.482 |
| WALL-X | 4 | 4224.039 |
| GR00T | 32 | 1068.806 |
EgoForce uses a four-frame history, while the eight external baselines use single-frame input. Section 3 compares four-frame GRU, four-frame pooling, and single-frame EgoForce to examine aggregation method and history length. Section 4 evaluates the expert architecture through a parameter-matched residual-head comparison with the same backbone, batch size, and four-frame history.
- Response to W2 and Q2: Expert branch, GRU, observation history, and confidence intervals for improvements
3.1. Expert branch, GRU, and observation history
The full model, the model without the expert branch, and the four-frame pooling model all use four observations. The GRU/pooling comparison fixes the observation length and compares aggregation methods. The single-frame model uses the current RGB image and hand state to quantify the benefit of observation history. Every configuration below uses three seeds and 10,000 training steps per seed.
ID: Three-seed results for the relevant ablations
| Model | Contact MAE (N) | Contact | Contact Pearson | Contact |
|---|---|---|---|---|
| EgoForce (full) | ||||
| Without expert branch | ||||
| Four-frame pooling / without GRU | ||||
| Single-frame RGB + hand state |
Response to Reviewer 5RD8: Phase-Specific Evaluation, Statistical Robustness, and Dataset Clarifications - Part 1/6
Thank you for recognizing FEEL's controlled physical conditions, continuous fingertip-force supervision, and multidimensional evaluation. To address your concerns, we report pre-contact and phase-specific evaluations, three-seed comparisons with all baselines, targeted component ablations, and detailed data splits and contact-score analyses for the current dataset. Learned-model results on the fixed Train/ID/OOD split are reported as mean ± sample standard deviation across three random seeds. We additionally report 95% confidence intervals for the paired improvements from the expert branch. The independent cross-demonstrator evaluation is described separately in Section 5 of this response.
We evaluate three stages separately: before contact, initial contact, and stable grasping. At each stage, we compare full RGB-plus-hand-state input, hand-state-only input, and the training-set per-finger mean-force reference.
1.1. Strict pre-contact prediction
Let t0 be the first frame in an episode at which the ground-truth force of either finger exceeds 0 N. The model receives four consecutive actual observation frames from t0−4 through t0−1 and uses the output for this window at t0−1 to predict the two-channel fingertip force at t0. All inputs precede first contact. At 10 Hz, the final input and the target are approximately 0.1 s apart.
This evaluation directly uses the existing step-10,000 checkpoints and their corresponding normalization parameters. The full model and the hand-state-only model use observation histories of the same length. The training-set mean-force reference uses the per-finger means from the training data. Evaluation covers 302 ID episodes and 113 Empty Milk OOD episodes. MAE is first averaged over the two force channels and then averaged equally across episodes.
On Empty Milk OOD, the full model achieves a mean MAE of 1.349 N, a 76.5% reduction from 5.739 N for hand-state-only input; the training-set mean-force reference yields 14.567 N. On ID, the full model achieves 3.662 N, compared with 3.976 N for hand-state-only input and 13.609 N for the training-set mean-force reference. This restriction on input timing directly tests the ability of pre-contact observations to predict first-contact force and quantifies the contribution of visual information under the hidden-load condition.
1.2. Initial contact and stable grasping
Initial contact is defined as the three sampled frames starting at t0, namely t0, t0+1, and t0+2, corresponding to approximately [0,0.3) s after contact. Stable grasping uses the existing force-plateau label, phase=2, requires at least one finger to remain in contact, and excludes the initial-contact window. The MAE of the two fingertip-force channels is computed separately for each phase: we first average over that phase's frames and force channels within each episode, and then average equally across episodes.
Initial contact and stable grasping cover 871 and 787 ID episodes, respectively, and 261 and 253 OOD episodes, respectively.
During initial contact, the full model reduces mean MAE relative to hand-state-only input by 6.3% on ID and 61.3% on Empty Milk OOD. During stable grasping, the respective reductions are 29.1% and 48.8%. The full model also has lower mean error than the training-set mean-force reference in both phases on both sets, demonstrating the predictive value of visual information during early loading and sustained contact.