acceptodds
Under review as a conference paper at ICLR 2027

VidForce: Predicting Force-Aware Articulation by Observing Humans

Abstract

People can anticipate how an object moves and the effort an interaction may require by watching others. How can embodied agents acquire this understanding from video? Force measurements provide direct sensory supervision, but require specialized equipment and remain limited in quantity. We introduce VidForce, which estimates an interaction trajectory together with a force profile from a third-person RGB video; 2D contact clicks and textual object–action prompts are optional. Our approach makes limited instrumented measurements useful for interpreting human interactions through learned physical prototypes as a prior. The prototypes share learned force levels and temporal profiles across interactions of comparable effort, with visual and semantic cues determining their contribution to each observation. Geometric and vision–language features adapt this prior to the observed interaction, while a joint conditional flow refines an initial tracked trajectory together with the force profile. Paired human and instrumented-gripper recordings provide measured force references for supervising human video, and a separately trained torque model demonstrates that the formulation extends to other sensory quantities. We evaluate trajectory recovery and force estimation through baseline comparisons and controlled ablations on held-out interactions from the Hoi! dataset. On this evaluation subset, VidForce achieves 6.6 cm ATE, peak-force MAE and RMSE of 4.81 N and 5.92 N, respectively, and force-profile RMSE of 4.95 N. On real-world videos, VidForce infers motion and force reliably and consistently despite the variability in human execution. We further deploy VidForce on a Spot robot, where trajectories and force profiles estimated from human demonstrations drive realworld articulated-object manipulation, and report success rates across repeated trials. Project Page: https://vid-force.github.io/

Primary area:
applications to robotics, autonomy, planning
Keywords:
force estimation, human-object interaction, articulated objects, learning from human video, embodied AI, robotic manipulation, trajectory estimation, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7q94OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VidForce, a method for predicting 3D contact trajectories and force profiles from third-person videos of articulated-object interactions. It uses D4RT, Depth Anything 3, and Qwen3-VL to extract motion, geometry, and semantic features. A set of learned force prototypes provides an initial force estimate, which is adapted to the input video and refined together with the trajectory using a conditional flow model. The method is evaluated on the Hoi! dataset, where it achieves 6.6 cm trajectory ATE, 4.81 N peak-force MAE, and 4.95 N force-profile RMSE. The paper also includes real-world video experiments and Spot robot trials.

Critical strengths
  1. The paper addresses an interesting problem of predicting both contact trajectories and force profiles from human interaction videos. The improvement over D4RT initialization is substantial, and the force prediction results on instrumented recordings are encouraging.

  2. The experiments cover a good range of comparisons and ablations. The Spot experiments are interesting, particularly the improvement from 0/10 to 7/10 successes when using force feedforward for oven-door opening.

Critical weaknesses

I am not fully convinced by the contribution of the learned force prototypes. The categorical prior performs almost identically in Tab. 27. The simpler object–action category mean also performs better on drawers and is competitive on cupboard doors (Tab. 17), which account for most of the measured evaluation set. Much of VidForce's overall advantage comes from the oven-door and “Other” categories, with only six measured sequences between them. This makes it difficult to assess how well the model captures force differences within the same object–action category.

Questions for authors

The object–action category mean performs better on drawers and is competitive on cupboard doors. How well does VidForce capture force differences between interactions within the same category? How much of the improvement comes from a few high-force interactions?

The learned prototypes perform similarly to the categorical prior. What is the main benefit of using learned prototypes here? Given the similar performance with five and ten prototypes, is there evidence that the additional prototypes capture distinct force patterns?

Both instance adaptation and joint flow can modify the prototype force levels and temporal shapes. Could different prototype assignments lead to similar final predictions? How much of the predicted force profile actually comes from the prototypes rather than the subsequent corrections?

What motivates mixing prototype force levels in log space rather than directly in Newtons? Does this choice improve prediction accuracy compared with arithmetic interpolation?

How much do force predictions depend on the text prompts? Several refrigerator-closing interactions are interpreted as opening, substantially changing the routed force prior. How does the model handle incorrect action prompts, particularly when they contradict the observed motion?

Force prediction on human videos is considerably less accurate than on gripper videos, and GPT-5.5 achieves lower peak MAE on the human subset. What accounts for this gap?

The evaluation crops are based on annotated articulation extent, and the self-click setting retains the annotated contact depth. How much do these choices affect trajectory accuracy? Which validation data were used to choose the prototype count and teacher-switching step?

The adapted prior already achieves 4.84 N peak MAE, compared with 4.81 N after joint-flow refinement. What does joint flow contribute to force prediction beyond the adapted prior?

The predicted peak force can differ from the maximum of the predicted force profile. How large is this discrepancy in practice, and which value is used during robot execution?

VidForce uses several large frozen backbones in addition to its trainable parameters. What are the total inference cost and latency compared with the baselines?

The real-world experiments include luggage-scale measurements, but the agreement with the predictions is not quantified. Could the authors report these errors and clarify the near-zero measurement markers?

In the Spot experiments, would a constant or category-based force feedforward give similar results? Also, the description of the controller stiffness appears inconsistent across conditions. Were the stiffness and damping gains kept the same?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 7q94 (1/7)

Thank you for your careful review and detailed questions. We address them below with expanded ablations to answer each of them, along with additional robot experiments and simulations.

W: The learned prototypes achieve comparable accuracy to the categorical prior without requiring ground-truth category annotations at training or inference time (Table 4; Appendix C.11, Table 38). Their benefit also extends to data efficiency: with half the force labels, VidForce reaches 5.2 N peak MAE, compared with 5.9 N for a global prior trained with all labels (Appendix C.6, Table 28). The details are in Q2, Q3 and Q8.

We have expanded Table 4 to make the contribution across force ranges explicit. Compared with the category mean, VidForce reduces peak MAE from 4.6 to 4.2 N below 20 N and from 12.5 to 7.3 N at or above 20 N. The strongest gains are therefore on high-force interactions, while VidForce achieves the lowest overall mean peak MAE, tail-object error and profile RMSE among the evaluated priors.

For variation within a category, the routed prior provides an initial estimate that is adapted to the observed interaction. Instance adaptation reduces peak MAE from 6.9 to 4.8 N (Table 3), allowing interactions sharing a prior to receive different predictions. We have also added an analysis within each object–action group across execution speeds (Appendix C.14, Tables 44 and 45), where the error does not rise steadily with speed. Q1 gives the per-object numbers.

Our goal is to learn from limited sensory data. Our analysis shows that the prototypes provide an effort-based starting estimate, the instance adaptation adjusts its magnitude and shape to the observed interaction, and the joint flow refines the trajectory and corrects the remaining force error when the prior is weak (Tables 1 and 3). The stages therefore complement one another, reducing the sensitivity to the exact number of prototypes (Appendix C.8, Table 32) while supporting predictions across low-force, high-force and tail interactions (Table 4). Their combined benefit also shows in the full temporal force profile, where VidForce achieves the lowest RMSE among the evaluated priors (Table 4). We acknowledge data availability is limitation collecting more diverse data, curating one with better distribution fidelity and variance in physical properties in focus of our future efforts.

AuthorsOpenReview ↗

Response to Reviewer 7q94 (2/7)

Q1. How well does VidForce capture force differences within a category, and how much of the improvement comes from a few high-force interactions?

Thank you, this question highlights the strength of VidForce well, we have expanded Table 4 in main paper to show to show this properly Comparable to the category mean, without privileged labels and better on tail The category mean is given the ground-truth object–action label of each test sequence to our model. VidForce learns its prototypes and routing from the video and the prompt. On the drawer, cupboard-door and refrigerator-door sequences, the two are comparable (4.56 N for VidForce, 4.29 N for the category mean; Appendix C.5, Table 25). VidForce is better on cupboard doors, and the category mean on drawers.

VidForce has lower error on both high-force and normal-force interactions. Against the category mean, it has 4.2 against 4.6 N on the sequences below 20 N, and 7.3 against 12.5 N on the sequences at 20 N and more (Table 4; Appendix C.5, Table 26). The category mean predicts at most 45 N, while VidForce predicts 58 N for the oven doors, which are measured at about 70 N (Table 26).

Why? This follows from the formulation: the prototypes hold several force levels, organized by measured effort and not by category. The same model with one global prior instead of prototypes is equally accurate on the light interactions, but has 11.8 N error on the 9 high-force sequences (Table 4). For differences within a category, we also split each object–action group by execution speed (Appendix C.14): within a group, the error does not rise steadily with speed (Tables 44 and 45). The new tables are in Appendix C.5.

Q4. What motivates mixing prototype force levels in log space rather than directly in Newtons? Does this choice improve prediction accuracy compared with arithmetic interpolation?

Why log space. Force levels differ by ratios, not by fixed amounts: an error of 2 N is large for a 5 N drawer and small for a 70 N oven door. Mixing the levels in log space therefore seemed a sensible choice to us. It also keeps the model on one scale, because the prior is trained on the log peak, and the adaptation and the flow both apply multiplicative corrections (Sec. 3).

Does it improve the accuracy? It does not matter: both give almost the same error. We switched the trained checkpoints to arithmetic mixing, keeping all weights. Peak MAE is 4.81 N with log mixing and 4.83 N with arithmetic mixing, and peak RMSE is 5.92 and 5.95 N, because a typical sequence mixes levels that are close in value. We have added this comparison to Appendix C.8. Given this small difference, we do not see a reason to retrain the model with arithmetic mixing, but we would be happy to do so if the reviewer would like to see it.

Q5. How much do force predictions depend on the text prompts? Several refrigerator-closing interactions are interpreted as opening, substantially changing the routed force prior. How does the model handle incorrect action prompts, particularly when they contradict the observed motion?

How much does the force depend on the prompt? The dependence is limited. As a worst case, we flipped the action word (open ↔ close) in every prompt of the evaluation set. Peak MAE rises only from 4.8 to 5.5 N (Appendix C.13, Table 41). The force is affected because the prompt enters the routing of the prior. Even with every action wrong, VidForce stays more accurate than the category mean with the correct label (6.1 N). In normal use, the generated prompt reads the action correctly in 93% of the evaluation sequences. This means that a wrong prompt cannot override what the model reads from the video, while a fitting prompt helps.

What happens to the trajectory when the prompt contradicts the motion? Nothing much. The motion is read from the video, not from the prompt. With all action words flipped, the trajectory error changes only from 6.6 to 7.0 cm (Table 41). On the real-world videos, the direction of the predicted trajectory follows the demonstrated action in all 46 episodes, including the 10 where the generated prompt is wrong (Appendix D.1). The prompt is an input, so a human or a stronger vision-language model can supply it and avoid these misreads.

AuthorsOpenReview ↗

Response to Reviewer 7q94 (3/7)

Q6. Force prediction on human videos is considerably less accurate than on gripper videos, and GPT-5.5 achieves lower peak MAE on the human subset. What accounts for this gap?

Human videos are scored against proxy labels. Hoi! has no force sensor on the hand, so the reference is the force measured in the gripper execution of the same episode, not the force of the hand.

On GPT-5.5, at first we thought it simply understands human videos better. It does not look that way. Its advantage is limited to the force peak on human videos, where its error is lower than ours by 2.7 N. On the gripper videos of the same episodes VidForce is lower by 2.8 N, and on torque VidForce is lower for both embodiments. So VidForce is ahead in three of the four cases. GPT-5.5's error is also nearly the same for the human and the gripper video, which indicates that the GPT prediction is mostly guided by the object instead of motion or embodiment. Note that GPT-5.5 returns one peak value only, while VidForce also predicts the contact trajectory and the force profile over time (Appendix C.10, Table 37).

ModelForce, gripper video: MAEForce, gripper video: RMSEForce, human video: MAEForce, human video: RMSETorque, gripper video: MAETorque, gripper video: RMSETorque, human video: MAETorque, human video: RMSE
GPT-5.5 (8 frames)7.9813.117.9513.720.6750.8290.6300.788
GPT-5.5 (8 frames + text)8.0912.598.3213.090.5710.7850.6270.827
VidForce5.156.4010.6414.090.3180.4000.3560.453
Δ VidForce vs GPT-5.5-2.8-6.2+2.7+1.0-0.25-0.38-0.27-0.33

Peak errors, force in N and torque in N m. Δ is VidForce minus the better GPT-5.5 row of the column, so a negative value means that VidForce has the lower error.

Q7. The evaluation crops are based on annotated articulation extent, and the self-click setting retains the annotated contact depth. How much do these choices affect trajectory accuracy? Which validation data were used to choose the prototype count and teacher-switching step?

How much do the crop and the contact depth affect trajectory accuracy? Not much. Removing either annotation changes ATE by at most 0.4 cm (Appendix C.13, Table 42). With a fixed crop instead of the annotated extent, ATE goes from 6.6 to 7.0 cm. With the depth read at the predicted contact pixel instead of the annotated one, it goes from 7.0 to 7.1 cm. Fully automatic, with no annotation at all, VidForce reaches 7.1 cm ATE and 4.5 N peak MAE. It is still better than all the baselines on aligned and unaligned error (best baseline 11.8 cm ATE and 20.3 cm ADE; Tables 1 and 18). Appendix C.3 shows the effect of shifted, rescaled and mirrored crops (Fig. 8).

Which validation data chose the prototype count and the teacher step? Neither was tuned for the reported accuracy. K = 10 was approximately set from the statistical insights of Hoi Dataset, and the model is insensitive to K (see Q2). In practice, the teacher step was fixed once, early in development for full MLP (Table 4) based on held out sequences from the train scenes, at the point where the force error on sequences held out from the training set stops improving (Appendix A.4). Every model and variant later uses the same step.

Individual unseen scenes, including two new ones. For further evidence, the authors of Hoi! graciously provided two new scenes that are pending release and were completely inaccessible to us during development. Appendix C.4 (Table 22) reports ATE, peak MAE and peak RMSE on each individual unseen scene, with the frozen checkpoints: the new scenes overall confirm the robustness, we inspected the results for livingroom_2 further and found that the higher ATE comes from cabinet doors with glass inlets, which affect tracking and motion fields.

Unseen sceneSequencesATE (cm)Peak MAE (N)Peak RMSE (N)
livingroom_1476.74.775.67
livingroom_2 (new)3511.86.427.43
kitchen_6 (new)214.74.507.02
All three combined1037.15.116.48
Reviewer_Jnh6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VidForce jointly predicts a 3D contact trajectory and a force-magnitude profile from a static third-person interaction video, with optional contact and language guidance. Frozen motion, depth, and vision–language models provide features to a trainable fusion network. Learned effort prototypes initialize the force estimate, instance adaptation adjusts that prior, and a conditional flow jointly refines trajectory and force. Training uses Hoi! instrumented-gripper measurements and down-weighted references transferred to paired human recordings. The evaluation contains 99 sequences, of which 48 gripper sequences have measured force. Across three training seeds, VidForce obtains 6.6 ± 0.4 cm median trajectory error and 4.81 ± 0.38 N peak-force MAE on measured gripper sequences. Extensive controls examine priors, routing, refinement, and limited labels. Separate studies investigate human-reference agreement, real-world videos, torque prediction, and manipulation on Spot.

Critical strengths

A key strength is the useful formulation of jointly recovering a 3D interaction trajectory and a temporal force profile from ordinary video, extending visual imitation with information relevant to physical execution. The learned effort prototypes offer a practical way to exploit scarce force measurements: with half the labeled training episodes, VidForce achieves lower peak-force error than the full-data global-prior variant (5.23 versus 5.61 N). A second strength is the comparatively thorough experimental support for the modeling choices. Across three training seeds, parameter-matched prior comparisons and controlled refinement ablations substantiate the benefits of the prototypes and joint trajectory–force prediction. Trajectory error decreases from 11.8 cm for the tracked initialization to 6.6 cm, while the prototype model reduces measured-gripper peak-force MAE from 5.79 N for a parameter-matched MLP prior to 4.81 N. The Spot deployment additionally demonstrates that the predicted trajectories and force profiles can be incorporated into real manipulation, giving the approach practical relevance beyond offline estimation.

Critical weaknesses

The main weakness is that force inference from human video remains indirectly validated. The headline 4.81 N peak-force MAE is measured on instrumented-gripper recordings, while human videos receive reference forces from separate paired gripper executions. Differences in contact geometry, friction, and execution speed can affect both force magnitude and temporal shape, so agreement with these references does not establish accurate recovery of the effort in the observed human interaction. This limits the interpretation of the paper’s central transfer claim. A second weakness is that the robot experiment does not clearly isolate the benefit of the predicted temporal force profile. Appendix D.3 states that controller gains are identical between conditions, but also describes reducing stiffness along the motion direction when predicted force is applied, leaving a potentially consequential control difference unclear. Moreover, the experiment covers only two objects and lacks constant-force or object/action-mean-force controls; the observed success improvement is concentrated on the oven door. Clarifying the controller settings and including these controls would help determine whether the benefit comes from the learned force profile or simply from supplying additional force. These issues constrain confidence in the physical interpretation and deployment benefits, despite the strong offline ablations.

Questions for authors
  1. Clarification—meaning of force predictions from human video. Are these outputs intended to estimate the force actually exerted by the human, or the force a gripper would require for the corresponding interaction? Please clarify the assumptions behind transferring gripper measurements to human recordings and how the existing evaluation supports the intended interpretation.

  2. Clarification—robot controller comparison. Appendix D.3 states that both conditions share identical gains, but also describes reduced path-direction stiffness when predicted force is applied. Were stiffness and damping identical between conditions? Please provide the settings needed to determine whether the success improvement can be attributed to the feed-forward force term.

  3. Suggested additional validation—benefit of the temporal profile. Under matched controller gains and force limits, how does the predicted profile compare with constant or object/action-mean feed-forward force? This comparison would help establish whether the oven-door improvement depends on the learned profile’s instance-specific temporal structure.

If the first two points get sorted out convincingly, and especially if the third one gets some supporting evidence, that'd go a long way toward convincing me on the transfer and deployment claims — and I could see raising my rating.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Jnh6 (1/2)

Thank you for your thoughtful review and positive assessment. We address your concerns below by clarifying the interpretation of the predicted force and presenting additional comparisons under matched controller settings to examine the benefit of the temporal force profile.

Wa: As clarified in Q1, VidForce predicts the force a robot gripper would need for the observed interaction, not the force exerted by the human hand, which would need hand-worn force sensors; the introduction and the method now state this clearly (Sec. 1, Sec. 3.4). For this target, the instrumented recordings are a directly matched comparison: in Hoi!, a person operates a hand-held, instrumented gripper, so the force is measured on the same execution that the model sees. We also agree that the execution speed and the friction of the object change the force. On speed, we added a dedicated analysis (Appendix C.14, Tables 43 to 45): within an object–action group, the error does not rise steadily with speed, so the larger error of fast executions reflects the objects they contain, such as the oven doors. On friction and resistance, our force sensors are still in procurement, and we could not find public data for this; we had stated it as a limitation (Sec. 4.4) and will focus our future efforts on curating such a dataset.

Wb. Please see Q2, where we address both points with new experiments. Controller: all conditions now use exactly the same configuration (Appendix D.2). Controls: on Spot, we compare the predicted profile with no force and with a constant force under identical settings (Table 6), and calibrate the constant force step by step (Table 47); the object/action mean and a temporally shuffled profile are compared in simulation (Table 50). The improvement concentrates on the oven door because it is the high-effort object: the drawer opens from the trajectory alone, even when loaded, while the oven door stays shut without force and under a constant 20 N, and opens with the predicted profile. Supplying additional force can also open it, but it has to be calibrated for each object to work safely: a constant 60 N quickly reaches our safe limit on Spot (Fig. 4), and in simulation a constant at the peak force reaches the end stop with a much higher impact energy (Table 7). On the limited number of objects, our experiments with the dishwasher, the fridge were not approved, because similar experiments led to damage to the robot in the past.

AuthorsOpenReview ↗

Response to Reviewer Jnh6 (2/2)

Q1: Clarification—meaning of force predictions from human video. Are these outputs intended to estimate the force actually exerted by the human, or the force a gripper would require for the corresponding interaction? Please clarify the assumptions behind transferring gripper measurements to human recordings and how the existing evaluation supports the intended interpretation.

The force a gripper would require. The purpose is not to learn the human force, but to learn the relation between the visual cues of a human interaction and the force profile a robot would need (Sec. 1, Sec. 3.4). Just as a human who watches another human open an oven can estimate the trajectory and the effort, we want to give a robot the same ability. To measure the human force exactly, hand force sensors are the better instrument. Our target is a method for robotics, so all our evaluations and experiments are designed to test the robotics use case (Sec. 4.3, Table 6; Appendix D.2, D.3). The approach also works on different sensor modalities, force and torque (Table 2; Appendix C.12). We have now revised the introduction and the method to state this clearly (Sec. 1, Sec. 3.4).

Q2: Clarification—robot controller comparison. Appendix D.3 (now App. D.2) states that both conditions share identical gains, but also describes reduced path-direction stiffness when predicted force is applied. Were stiffness and damping identical between conditions? Please provide the settings needed to determine whether the success improvement can be attributed to the feed-forward force term.

Q3: Suggested additional validation—benefit of the temporal profile. Under matched controller gains and force limits, how does the predicted profile compare with constant or object/action-mean feed-forward force? This comparison would help establish whether the oven-door improvement depends on the learned profile's instance-specific temporal structure.

Extended and matched robot trials. We have extended our trials, with identical configurations fully explained in Appendix D.2. We conducted our trials with no force, a constant force and our predicted force profile (Table 6). For the constant force, we did a controlled calibration (Table 47). As a constant force at contact can lead to excessive forces [1] as the tracking initially can add addition force on top of feed-forward, which can damage the robot (a natural approach toward this is to relatively reduce stiffness in direction of motion), hence we set a safe limit of 1.3 times the maximum planned force, which stops the Spot gripper.The results of the trials and the experimental setup are discussed in detail (Sec. 4.3, Fig. 4; Appendix D.2, Tables 47 and 48). Conclusion. (1) VidForce produces robot-executable trajectories, and (2) its force profile enables safe articulation by the robot. A constant-force controller can be designed, but it has to be calibrated case by case: a constant 60 N opened the oven door, but quickly reached our set safe limit. These experiments are also aligned with concurrent work [1] (Sep 16, 2026), which shows the importance of a force profile. However, unlike them, we operate in small as well as high force ranges, which creates another issue: the kinetic (impact) energy. We investigated this in an additional simulation study of the predicted profile against constant forces, the category mean and a temporally shuffled profile (Table 50).

Simulation study of force. We simulate the oven door and drawer in MuJoCo 3.4 with 0.2 ms steps, using joint friction, position-dependent latch resistance and end stops (Appendix D.3, Fig. 17). For the hand-built objects, the resistance is set manually, based on the literature [2]. The robot is represented by the controller forces applied at the handle, using the interpolated 32-sample profile. Success requires 95% of the travel within the execution window plus 1.5 s. The impact energy is the panel's kinetic energy just before end-stop contact, or 0 J without contact. Force profiles produce a controlled opening with a lower impact. The progress-based profile opens all three oven-door twins at a lower impact energy than a constant at the predicted peak; a constant at the category mean fails to open them. Drawer profiles also handle empty, 3.4 kg and 35 kg loads without hitting the stop (Table 49, Fig. 18). Constant forces require calibration for each object to achieve comparable control [3, 4].

[1] G. Ji et al., “Dreaming the sound of contact: Leveraging video and audio generation for zero-shot force-aware manipulation and data generation,” arXiv:2609.19137, 2026.

[2] D. J. Schweers et al., “Frictional drawer slide dampener,” U.S. Patent 7,946,667, 2011.

[3] A. Jain et al., “The complex structure of simple devices: A survey of trajectories and forces that open doors and drawers,” BioRob, 2010.

[4] A. Jain et al., “Improving robot manipulation with data-driven object-centric models of everyday forces,” Autonomous Robots, 2013.

Reviewer_GRF2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VidForce estimates a 3D contact trajectory and a temporal force-magnitude profile from third-person RGB video, optionally using a contact click and text prompt. It combines frozen visual backbones, learned effort-based prototypes, and joint conditional flow refinement. Paired gripper recordings provide force supervision for human videos. Evaluation covers Hoi!, additional real-world videos, and articulated-object manipulation with Spot.

Critical strengths
  1. Relevant and useful problem. Recovering interaction effort alongside motion could make human demonstrations more informative for robot learning. This is well aligned with CVPR’s interests in physical scene understanding.

  2. Thoughtful architecture and ablations. Separating force magnitude from temporal shape is reasonable. Table 4 includes meaningful controls: a parameter-matched MLP prior, categorical priors, one-shot regression, and separate trajectory/force flows.

  3. Encouraging empirical evidence. The method improves trajectory recovery over its tracking initialization and force estimation over the evaluated third-person baselines. The reduced-supervision study supports the usefulness of learned prototypes when measurements are scarce.

  4. Generally clear presentation. The pipeline and objectives are understandable, and the paper explicitly acknowledges several limitations, including monocular scale uncertainty and the use of gripper-derived human supervision.

Critical weaknesses
  1. The central human-video claim lacks independent force validation. Human recordings inherit force profiles from separate gripper executions. Differences in speed, contact location, and execution strategy can change both magnitude and timing. Consequently, the human-video ablation demonstrates improved agreement with proxy labels, rather than accuracy against the force actually exerted during the observed interaction. The main force benchmark primarily establishes performance on instrumented-gripper recordings.

  2. The novelty framing overlooks directly relevant work. Section 4.1’s statement that no prior method jointly predicts trajectory and force is too broad: Li et al. [1] jointly estimated motion and interaction forces from monocular video. EgoPHI [2] also estimates contact and forces for articulated objects, although it uses egocentric images and known object geometry. These differences leave room for VidForce’s contribution, but the novelty should be framed around its prototype prior, supervision transfer, and particular prediction setting. ZM

  3. Instance-specific physical generalization remains uncertain. The categorical prior performs comparably to learned prototypes in Table 4, while Section 4.3 reports that most opening interactions receive the same routed prior. Figure 3 also shows substantial overestimation relative to the luggage-scale readings for the cabinet and drawer. These observations suggest that reliable prediction beyond object–action expectations needs stronger evidence.

  4. The robot study does not isolate the value of the predicted profile. Adding predicted force improves oven opening, which is promising. However, comparison against no feed-forward force cannot establish whether the learned temporal profile is necessary: a constant or category-conditioned force might provide similar assistance. The small set of manipulated objects further limits the conclusion.

  5. The automatic-inference evaluation is incomplete. Section 4.1 states that cached self-click experiments retain depth at the annotated contact. Moreover, start-aligned trajectory error removes absolute initial-position error. Together, these choices leave important localization and depth failures untested. The visibility-filtered evaluation also provides limited evidence for difficult, uncurated videos.

[1] Li et al. Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video. CVPR, 2019.

[2] Ilic et al. EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Vision. ECCV, 2026.

Questions for authors
  1. Add a modest test set with force measurements synchronized to the observed human executions. Include repeated interactions with varied speed and resistance, and report errors within each object–action group.

  2. Evaluate fully automatic contact localization and depth estimation, reporting both aligned and unaligned trajectory errors. Separate unseen-scene results from held-out-object results.

  3. Compare robot execution using predicted, constant, category-average, and temporally shuffled force profiles. Clarify the controller settings: Appendix D.3 mentions reduced along-path stiffness while also stating that both conditions use identical gains.

  4. Release the curated split identifiers and preprocessing pipeline, and report total inference cost across the frozen backbones and flow refinement. Distinguish prediction repeatability from measured physical accuracy in the real-world discussion.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GRF2 (1/3)

Thank you for your detailed review and for recognizing the relevance of the problem and the value of our ablations. Your comments helped us clarify the intended force prediction target and strengthen the evaluation. Below, we address each concern with clarifications and additional experiments, including fully automatic inference and matched robot trials.

Q1: Add a modest test set with force measurements synchronized to the observed human executions. Include repeated interactions with varied speed and resistance, and report errors within each object–action group.

Synchronized force. The gripper recordings of our evaluation set are such a test set: in Hoi!, a person operates a hand-held, instrumented gripper, so the force is measured on the same execution that the model sees. As explained in W1, we are interested in the force a robot would need, not in the force of the human hand, which would need hand-worn sensors (Sec. 1, Sec. 3.4).

Varied speed and repeated interactions in object-action group. We added a detailed analysis to Appendix C.14. We split the 48 measured recordings into slow, medium and fast executions (Table 43). The peak force error is 3.7, 4.5 and 6.1 N, against 4.8 N overall. The fast range contains all oven doors, which have the largest errors. The errors within each object–action group are reported in Tables 44 and 45: for drawers and cupboard doors, opening and closing, the peak force error lies between 3.3 and 5.0 N and shows no trend with speed inside a group. By going through the evaluation set by hand, we also found repeated interactions with the same object: the drawers of one sideboard are opened and closed 6 times each, and its doors 4 times each, at different speeds and viewpoints (Figs. 10 and 11).

Varied speed and resistance. Our force sensors are still in procurement, and we could not find public data for this. We agree that it is an important problem. We had stated it as a limitation (Sec. 4.4) and will focus our future efforts on curating such a dataset.

Q2: Evaluate fully automatic contact localization and depth estimation, reporting both aligned and unaligned trajectory errors. Separate unseen-scene results from held-out-object results.

Fully automatic contact and depth: re-evaluated (main paper, Tables 1 and 5; Appendix C.13, Tables 40 and 42). The contact is predicted by the model, the depth is read from DA3 at that predicted pixel/contact point, and the crop is fixed instead of taken from the annotation.

Aligned and unaligned errors: reported, now also in the main paper (Table 1). These are our ATE and ADE: ATE aligns the start of the trajectory and ADE does not. From the annotated contact to fully automatic inference, ATE goes from 6.6 to 7.1 cm and ADE from 11.4 to 17.0 cm, while the peak force error does not increase (4.8 to 4.5 N). The small increase in ATE compared to ADE indicates that the trajectory shape is kept and the error mainly comes from a wrong start position. Both errors are also reported for the baselines (Tables 1 and 18). On both metrics, VidForce is still better than all the baselines, also when it runs fully automatically.

Unseen scenes and held-out objects: reported separately, with two new unseen scenes. We asked the Hoi! authors for a pending release of additional scenes that had not been published so far. It adds two unseen scenes that were completely inaccessible to us during development (Appendix C.3, Table 21; Appendix C.4, Table 22). While the new scenes overall confirm the robustness, we inspected the results for livingroom_2 further and found that the higher ATE comes from cabinet doors with glass inlets, which affect tracking and motion fields.

Evaluation partitionSequencesATE (cm)ADE (cm)Peak force MAE (N)
Held-out interactions in seen scenes526.612.04.84
Unseen scene (livingroom_1)476.711.04.77
New unseen scene (livingroom_2)3511.817.76.42
New unseen scene (kitchen_6)214.712.74.50
All three unseen scenes1037.114.15.11
AuthorsOpenReview ↗

Response to Reviewer GRF2 (2/3)

Q3: Compare robot execution using predicted, constant, category-average, and temporally shuffled force profiles. Clarify the controller settings: Appendix D.3 (now Appendix D.2) mentions reduced along-path stiffness while also stating that both conditions use identical gains.

Extended and matched robot trials. We have extended our trials, with identical configurations fully explained in Appendix D.2. We conducted our trials with no force, a constant force and our predicted force profile (Table 6). For the constant force, we did a controlled calibration (Table 47). As a constant force at contact can lead to excessive forces [1] as the tracking initially can add addition force on top of feed-forward, which can damage the robot (a natural approach toward this is to relatively reduce stiffness in direction of motion), hence we set a safe limit of 1.3 times the maximum planned force, which stops the Spot gripper.The results of the trials and the experimental setup are discussed in detail (Sec. 4.3, Fig. 4; Appendix D.2, Tables 47 and 48). Conclusion. (1) VidForce produces robot-executable trajectories, and (2) its force profile enables safe articulation by the robot. A constant-force controller can be designed, but it has to be calibrated case by case: a constant 60 N opened the oven door, but quickly reached our set safe limit. These experiments are also aligned with concurrent work [1] (Sep 16, 2026), which shows the importance of a force profile. However, unlike them, we operate in small as well as high force ranges, which creates another issue: the kinetic (impact) energy. We investigated this in an additional simulation study of the predicted profile against constant forces, the category mean and a temporally shuffled profile (Table 50).

Simulation study of force. We simulate the oven door and drawer in MuJoCo 3.4 with 0.2 ms steps, using joint friction, position-dependent latch resistance and end stops (Appendix D.3, Fig. 17). For the hand-built objects, the resistance is set manually, based on the literature [2]. The robot is represented by the controller forces applied at the handle, using the interpolated 32-sample profile. Success requires 95% of the travel within the execution window plus 1.5 s. The impact energy is the panel's kinetic energy just before end-stop contact, or 0 J without contact. Force profiles produce a controlled opening with a lower impact. The progress-based profile opens all three oven-door twins at a lower impact energy than a constant at the predicted peak; a constant at the category mean fails to open them. Drawer profiles also handle empty, 3.4 kg and 35 kg loads without hitting the stop (Table 49, Fig. 18). Constant forces require calibration for each object to achieve comparable control [3, 4].

[1] G. Ji et al., “Dreaming the sound of contact: Leveraging video and audio generation for zero-shot force-aware manipulation and data generation,” arXiv:2609.19137, 2026.

[2] D. J. Schweers et al., “Frictional drawer slide dampener,” U.S. Patent 7,946,667, 2011.

[3] A. Jain et al., “The complex structure of simple devices: A survey of trajectories and forces that open doors and drawers,” BioRob, 2010.

[4] A. Jain et al., “Improving robot manipulation with data-driven object-centric models of everyday forces,” Autonomous Robots, 2013.

Q4: Release the curated split identifiers and preprocessing pipeline, and report total inference cost across the frozen backbones and flow refinement. Distinguish prediction repeatability from measured physical accuracy in the real-world discussion.

Split identifiers and preprocessing pipeline. We have now released the anonymized preprocessing pipeline and the identifiers of our evaluation set here: https://github.com/vid-force/vidforce-preprocessing (available during review). The full training and evaluation code and the checkpoints will be released on acceptance.

Total inference cost. We have now added Appendix A.6 (Table 13), which shows the inference cost of VidForce with a breakdown of each component, along with the inference cost of the baselines. One 32-frame episode takes 4.2 s on one GPU with a supplied prompt and contact, and 5.0 s fully automatic. The frozen backbones take almost all of this time (depth estimation alone 2.8 s), and the flow refinement takes 0.07 s.

Repeatability versus physical accuracy. We have now distinguished these in the real-world discussion (Sec. 4.3). Repeatability is the spread of the predictions over repeated openings of the same object, for example 41 to 46 N over the four oven-door openings (Fig. 3b). Physical accuracy is the difference to the luggage-scale reading: 6.6 N on average over the four objects where the scale captured a reading (Appendix D.1, Table 46).

AuthorsOpenReview ↗

Response to Reviewer GRF2 (3/3)

W1. The purpose is not to learn the human force, but the equivalent force a robot would need, because most robots today operate with a gripper. Just as a human who watches another human open an oven can estimate the trajectory and the effort, we want to give a robot the same ability. Measuring the human force exactly would need hand force sensors. Our target is a method for robotics, so all our evaluations and experiments test the robotics use case. The introduction and the method now state this clearly (Sec. 1, Sec. 3.4). The approach also works for torque (Table 2), and we further note that the accuracy gains in trajectory estimation are also very relevant for robotic deployment. The influence of the execution speed is covered in Q1.

W2. Thank you for pointing us to these papers, in particular the concurrent work EgoPHI [2], which was released on arXiv on Aug 13, 2026. Following your suggestions, both works are now discussed in Sec. 2, and Sec. 4.1 states the novelty more clearly and distinguishes it from these works. The key difference is what each method predicts. Li et al. [1] treat rigid, stick-like hand tools without articulation, through per-video trajectory optimization. EgoPHI [2] predicts contact and normal force for a single egocentric image with a known object mesh, trained on simulated forces.

W3. The categorical prior requires privileged ground-truth category information: with the category withheld at test time, its peak MAE rises from 4.9 to 8.2 N (Appendix C.11, Table 38). VidForce does not require category annotation at training or inference time. We have now expanded Table 4: VidForce works well overall and on the full profile, and also predicts well on the tail objects that the annotation bundles as "other" (more in Appendix C.5, Tables 25 and 26). The same routed prior does not mean the same prediction (Sec. 4.3): the prior only provides a starting point, a good initial prior essentially lowers the burden on the later adaptation and joint flow (Sec. 4.2, Table 3). The luggage scale showed no reading for the drawer and the cabinet door, because these light objects were opened relatively fast and the scale needs a few seconds to settle (Fig. 3; Appendix D.1; see also Q4). A drawer can also be empty or hold a heavy object, and in simulation, a force profile like ours opens it either way without hitting the stop (Table 7).

W4. Please see Q3. We conducted Spot trials with matched config and provided complementary study from simulation (Sec. 4.3). On Spot, we execute the predicted trajectories without force, with a constant force, and with the predicted force profile, under the same controller settings (Table 6). The drawer opens from the trajectory alone, even when loaded, and the force profile lets Spot open the high-effort oven door, which stays shut without force and under the constant force. A constant force calibrated for each object can also succeed, as on the drawer, but a constant 60 N opened the oven door relatively quickly and hit our safe limit (Fig. 4; Appendix D.2). In simulation, a force profile like ours opens the drawer and the oven door with low impact energy, while a constant pull at the peak force hits the end stop with a very high impact energy (Table 7; more in Appendix D.3).

W5. Clarification: we should have written this properly. Both self-click and manual click use the depth predicted by DA3. In the submitted version, it was written: self-click still sampled it at the annotated contact pixel. It now samples it at the predicted pixel. Thank you for pointing this out. We fixed the text (Sec. 4.1) and re-ran the numbers (Tables 1, 5 and 40). They change by 0.1 cm ATE and at most 0.08 N in peak force error, within the seed variation. The fully automatic evaluation with aligned and unaligned errors is in Table 1 and in Q2. The robot trials also log trajectory failures, which were mainly due to localization error (Table 6). Our method can take a 2D contact point as input, so a manual click or a dedicated contact model can mitigate them. Our study focuses on visible interactions, which we had stated as a limitation (Sec. 4.4). The real-world phone videos (Sec. 4.3) are outside Hoi! and are run with self-click and self-prompt.

[1] Z. Li, J. Sedlar, J. Carpentier, I. Laptev, N. Mansard, and J. Sivic, "Estimating 3D motion and forces of person-object interactions from monocular video," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019.

[2] A. Ilic, R. Schuchert, Y. Jiang, and C. Holz, "EgoPHI: Estimating 3D hand-object contact and force from egocentric vision," in Proc. European Conf. Computer Vision (ECCV), 2026.

Comments on the paper

2 comments

Dear Reviewers,

Thank you again for your reviews of this submission. The authors have now posted their responses, and the author–reviewer discussion period ends on November 18, 2026 (AoE).

Please read the authors' rebuttal and let them know whether it addresses your concerns. A short reply is enough, for example:

  • which of your concerns are now resolved;
  • which remain open, and what evidence or clarification would resolve them;
  • whether you plan to update your rating.

If the rebuttal changes your assessment, please update your review and rating before the discussion period ends. If you have already engaged, thank you, and no further action is needed unless the authors reply again.

Your participation is important for a fair and well-informed decision.

AuthorsOpenReview ↗

VidForce Rebuttal Summary

Dear Area Chair,

Thank you for overseeing the review. We also thank the reviewers for their valuable time and constructive feedback. We would like to summarize VidForce’s two primary contributions, the robot trials and the additional evidence provided during the discussion.

  • VidForce predicts articulation trajectories and force profiles from third-person human video for robotics applications. The intuition is that humans can watch someone interact with an object and estimate both how it moves and the effort needed. We want to give robots the same capability. Hence, we are interested in the forces a robot gripper would need, not the exact forces exerted by the human hand, as we clarified to reviewers GRF2 and Jnh6.

  • VidForce also proposes a method for effectively using limited sensory data, demonstrated on force and torque. We use Hoi!, which provides paired videos of a person interacting with an object using their hand or an instrumented gripper. The learned prototypes provide a starting estimate that is adapted to each interaction, reducing the correction left to the flow. With half of the force labels, VidForce reaches 5.2 N peak MAE, outperforming the global-prior variant trained with all labels. The prototypes also match the categorical prior without requiring its ground-truth category information and perform well on tail objects.

  • Our robot trials demonstrate that the predicted trajectories are usable for manipulation and that the force profiles support controlled execution. Reviewers GRF2, Jnh6 and 7q94 requested matched and extended trials to assess these claims and clarify the controller settings. We conducted additional Spot trials with identical stiffness and damping, comparing no feed-forward force, constant force and the predicted profile. The drawer opens from the trajectory alone, even when loaded. The oven door stays shut without feed-forward force and with a constant 20 N, while the predicted profile enables opening. A constant 60 N also opens it but quickly reaches the safety limit. We further highlight the problems with such a constant-force approach in simulation, where the predicted profiles reduce excess energy at impact compared with applying a constant force at the predicted peak. We also added comparisons against constant predicted peak, category mean, category peak and shuffled profiles, evaluating both opening success and impact energy.

  • Reviewer GRF2 requested synchronized force measurements from human interactions with varied speed and resistance. We clarified that we currently lack the required sensors and rely on public data; collecting such data is part of our future work. We also provided results split by execution speed using Hoi!’s instrumented-gripper recordings, where a person operates the gripper at different speed and force is measured during the same execution shown in the video.

  • We addressed the automatic-inference concerns of reviewers GRF2 and 7q94 by rerunning evaluation with predicted contact, depth sampled at that contact and a fixed crop. Fully automatic inference achieves 7.1 cm ATE and 4.5 N peak-force MAE.

  • Reviewer 7q94’s main concern was the contribution of the prototypes. Their detailed questions and requested experiments helped us expand the ablations, refine the manuscript and add details on the method and experimental settings to improve reproducibility. Reviewer 7q94 acknowledged that most concerns had been addressed.

  • Reviewer 7q94’s follow-up also prompted us to check profile scaling. We corrected tables that had reported unscaled profiles. The overall change in profile RMSE is below 0.6 N, with no change in relative rankings.

  • Reviewer Jnh6 indicated that clarifying the force target and controller settings, together with additional force-profile comparisons, could lead to a more positive assessment. We provided these clarifications and added matched robot trials and simulation controls in response.

  • To further support our findings, we asked the Hoi! authors for additional scenes pending release. We evaluated our frozen checkpoints on two such scenes and reported the results. These scenes were unavailable to us during development and therefore further highlight our model's generalization from limited data.

Thank you for considering our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.