- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VidForce, a method for predicting 3D contact trajectories and force profiles from third-person videos of articulated-object interactions. It uses D4RT, Depth Anything 3, and Qwen3-VL to extract motion, geometry, and semantic features. A set of learned force prototypes provides an initial force estimate, which is adapted to the input video and refined together with the trajectory using a conditional flow model. The method is evaluated on the Hoi! dataset, where it achieves 6.6 cm trajectory ATE, 4.81 N peak-force MAE, and 4.95 N force-profile RMSE. The paper also includes real-world video experiments and Spot robot trials.
Critical strengths
-
The paper addresses an interesting problem of predicting both contact trajectories and force profiles from human interaction videos. The improvement over D4RT initialization is substantial, and the force prediction results on instrumented recordings are encouraging.
-
The experiments cover a good range of comparisons and ablations. The Spot experiments are interesting, particularly the improvement from 0/10 to 7/10 successes when using force feedforward for oven-door opening.
Critical weaknesses
I am not fully convinced by the contribution of the learned force prototypes. The categorical prior performs almost identically in Tab. 27. The simpler object–action category mean also performs better on drawers and is competitive on cupboard doors (Tab. 17), which account for most of the measured evaluation set. Much of VidForce's overall advantage comes from the oven-door and “Other” categories, with only six measured sequences between them. This makes it difficult to assess how well the model captures force differences within the same object–action category.
Questions for authors
The object–action category mean performs better on drawers and is competitive on cupboard doors. How well does VidForce capture force differences between interactions within the same category? How much of the improvement comes from a few high-force interactions?
The learned prototypes perform similarly to the categorical prior. What is the main benefit of using learned prototypes here? Given the similar performance with five and ten prototypes, is there evidence that the additional prototypes capture distinct force patterns?
Both instance adaptation and joint flow can modify the prototype force levels and temporal shapes. Could different prototype assignments lead to similar final predictions? How much of the predicted force profile actually comes from the prototypes rather than the subsequent corrections?
What motivates mixing prototype force levels in log space rather than directly in Newtons? Does this choice improve prediction accuracy compared with arithmetic interpolation?
How much do force predictions depend on the text prompts? Several refrigerator-closing interactions are interpreted as opening, substantially changing the routed force prior. How does the model handle incorrect action prompts, particularly when they contradict the observed motion?
Force prediction on human videos is considerably less accurate than on gripper videos, and GPT-5.5 achieves lower peak MAE on the human subset. What accounts for this gap?
The evaluation crops are based on annotated articulation extent, and the self-click setting retains the annotated contact depth. How much do these choices affect trajectory accuracy? Which validation data were used to choose the prototype count and teacher-switching step?
The adapted prior already achieves 4.84 N peak MAE, compared with 4.81 N after joint-flow refinement. What does joint flow contribute to force prediction beyond the adapted prior?
The predicted peak force can differ from the maximum of the predicted force profile. How large is this discrepancy in practice, and which value is used during robot execution?
VidForce uses several large frozen backbones in addition to its trainable parameters. What are the total inference cost and latency compared with the baselines?
The real-world experiments include luggage-scale measurements, but the agreement with the predictions is not quantified. Could the authors report these errors and clarify the near-zero measurement markers?
In the Spot experiments, would a constant or category-based force feedforward give similar results? Also, the description of the controller stiffness appears inconsistent across conditions. Were the stiffness and damping gains kept the same?
10 comments
Response to Reviewer 7q94 (2/7)
Q1. How well does VidForce capture force differences within a category, and how much of the improvement comes from a few high-force interactions?
Thank you, this question highlights the strength of VidForce well, we have expanded Table 4 in main paper to show to show this properly Comparable to the category mean, without privileged labels and better on tail The category mean is given the ground-truth object–action label of each test sequence to our model. VidForce learns its prototypes and routing from the video and the prompt. On the drawer, cupboard-door and refrigerator-door sequences, the two are comparable (4.56 N for VidForce, 4.29 N for the category mean; Appendix C.5, Table 25). VidForce is better on cupboard doors, and the category mean on drawers.
VidForce has lower error on both high-force and normal-force interactions. Against the category mean, it has 4.2 against 4.6 N on the sequences below 20 N, and 7.3 against 12.5 N on the sequences at 20 N and more (Table 4; Appendix C.5, Table 26). The category mean predicts at most 45 N, while VidForce predicts 58 N for the oven doors, which are measured at about 70 N (Table 26).
Why? This follows from the formulation: the prototypes hold several force levels, organized by measured effort and not by category. The same model with one global prior instead of prototypes is equally accurate on the light interactions, but has 11.8 N error on the 9 high-force sequences (Table 4). For differences within a category, we also split each object–action group by execution speed (Appendix C.14): within a group, the error does not rise steadily with speed (Tables 44 and 45). The new tables are in Appendix C.5.
Q4. What motivates mixing prototype force levels in log space rather than directly in Newtons? Does this choice improve prediction accuracy compared with arithmetic interpolation?
Why log space. Force levels differ by ratios, not by fixed amounts: an error of 2 N is large for a 5 N drawer and small for a 70 N oven door. Mixing the levels in log space therefore seemed a sensible choice to us. It also keeps the model on one scale, because the prior is trained on the log peak, and the adaptation and the flow both apply multiplicative corrections (Sec. 3).
Does it improve the accuracy? It does not matter: both give almost the same error. We switched the trained checkpoints to arithmetic mixing, keeping all weights. Peak MAE is 4.81 N with log mixing and 4.83 N with arithmetic mixing, and peak RMSE is 5.92 and 5.95 N, because a typical sequence mixes levels that are close in value. We have added this comparison to Appendix C.8. Given this small difference, we do not see a reason to retrain the model with arithmetic mixing, but we would be happy to do so if the reviewer would like to see it.
Q5. How much do force predictions depend on the text prompts? Several refrigerator-closing interactions are interpreted as opening, substantially changing the routed force prior. How does the model handle incorrect action prompts, particularly when they contradict the observed motion?
How much does the force depend on the prompt? The dependence is limited. As a worst case, we flipped the action word (open ↔ close) in every prompt of the evaluation set. Peak MAE rises only from 4.8 to 5.5 N (Appendix C.13, Table 41). The force is affected because the prompt enters the routing of the prior. Even with every action wrong, VidForce stays more accurate than the category mean with the correct label (6.1 N). In normal use, the generated prompt reads the action correctly in 93% of the evaluation sequences. This means that a wrong prompt cannot override what the model reads from the video, while a fitting prompt helps.
What happens to the trajectory when the prompt contradicts the motion? Nothing much. The motion is read from the video, not from the prompt. With all action words flipped, the trajectory error changes only from 6.6 to 7.0 cm (Table 41). On the real-world videos, the direction of the predicted trajectory follows the demonstrated action in all 46 episodes, including the 10 where the generated prompt is wrong (Appendix D.1). The prompt is an input, so a human or a stronger vision-language model can supply it and avoid these misreads.
Response to Reviewer 7q94 (3/7)
Q6. Force prediction on human videos is considerably less accurate than on gripper videos, and GPT-5.5 achieves lower peak MAE on the human subset. What accounts for this gap?
Human videos are scored against proxy labels. Hoi! has no force sensor on the hand, so the reference is the force measured in the gripper execution of the same episode, not the force of the hand.
On GPT-5.5, at first we thought it simply understands human videos better. It does not look that way. Its advantage is limited to the force peak on human videos, where its error is lower than ours by 2.7 N. On the gripper videos of the same episodes VidForce is lower by 2.8 N, and on torque VidForce is lower for both embodiments. So VidForce is ahead in three of the four cases. GPT-5.5's error is also nearly the same for the human and the gripper video, which indicates that the GPT prediction is mostly guided by the object instead of motion or embodiment. Note that GPT-5.5 returns one peak value only, while VidForce also predicts the contact trajectory and the force profile over time (Appendix C.10, Table 37).
| Model | Force, gripper video: MAE | Force, gripper video: RMSE | Force, human video: MAE | Force, human video: RMSE | Torque, gripper video: MAE | Torque, gripper video: RMSE | Torque, human video: MAE | Torque, human video: RMSE |
|---|---|---|---|---|---|---|---|---|
| GPT-5.5 (8 frames) | 7.98 | 13.11 | 7.95 | 13.72 | 0.675 | 0.829 | 0.630 | 0.788 |
| GPT-5.5 (8 frames + text) | 8.09 | 12.59 | 8.32 | 13.09 | 0.571 | 0.785 | 0.627 | 0.827 |
| VidForce | 5.15 | 6.40 | 10.64 | 14.09 | 0.318 | 0.400 | 0.356 | 0.453 |
| Δ VidForce vs GPT-5.5 | -2.8 | -6.2 | +2.7 | +1.0 | -0.25 | -0.38 | -0.27 | -0.33 |
Peak errors, force in N and torque in N m. Δ is VidForce minus the better GPT-5.5 row of the column, so a negative value means that VidForce has the lower error.
Q7. The evaluation crops are based on annotated articulation extent, and the self-click setting retains the annotated contact depth. How much do these choices affect trajectory accuracy? Which validation data were used to choose the prototype count and teacher-switching step?
How much do the crop and the contact depth affect trajectory accuracy? Not much. Removing either annotation changes ATE by at most 0.4 cm (Appendix C.13, Table 42). With a fixed crop instead of the annotated extent, ATE goes from 6.6 to 7.0 cm. With the depth read at the predicted contact pixel instead of the annotated one, it goes from 7.0 to 7.1 cm. Fully automatic, with no annotation at all, VidForce reaches 7.1 cm ATE and 4.5 N peak MAE. It is still better than all the baselines on aligned and unaligned error (best baseline 11.8 cm ATE and 20.3 cm ADE; Tables 1 and 18). Appendix C.3 shows the effect of shifted, rescaled and mirrored crops (Fig. 8).
Which validation data chose the prototype count and the teacher step? Neither was tuned for the reported accuracy. K = 10 was approximately set from the statistical insights of Hoi Dataset, and the model is insensitive to K (see Q2). In practice, the teacher step was fixed once, early in development for full MLP (Table 4) based on held out sequences from the train scenes, at the point where the force error on sequences held out from the training set stops improving (Appendix A.4). Every model and variant later uses the same step.
Individual unseen scenes, including two new ones. For further evidence, the authors of Hoi! graciously provided two new scenes that are pending release and were completely inaccessible to us during development. Appendix C.4 (Table 22) reports ATE, peak MAE and peak RMSE on each individual unseen scene, with the frozen checkpoints: the new scenes overall confirm the robustness, we inspected the results for livingroom_2 further and found that the higher ATE comes from cabinet doors with glass inlets, which affect tracking and motion fields.
| Unseen scene | Sequences | ATE (cm) | Peak MAE (N) | Peak RMSE (N) |
|---|---|---|---|---|
| livingroom_1 | 47 | 6.7 | 4.77 | 5.67 |
| livingroom_2 (new) | 35 | 11.8 | 6.42 | 7.43 |
| kitchen_6 (new) | 21 | 4.7 | 4.50 | 7.02 |
| All three combined | 103 | 7.1 | 5.11 | 6.48 |
Response to Reviewer 7q94 (1/7)
Thank you for your careful review and detailed questions. We address them below with expanded ablations to answer each of them, along with additional robot experiments and simulations.
W: The learned prototypes achieve comparable accuracy to the categorical prior without requiring ground-truth category annotations at training or inference time (Table 4; Appendix C.11, Table 38). Their benefit also extends to data efficiency: with half the force labels, VidForce reaches 5.2 N peak MAE, compared with 5.9 N for a global prior trained with all labels (Appendix C.6, Table 28). The details are in Q2, Q3 and Q8.
We have expanded Table 4 to make the contribution across force ranges explicit. Compared with the category mean, VidForce reduces peak MAE from 4.6 to 4.2 N below 20 N and from 12.5 to 7.3 N at or above 20 N. The strongest gains are therefore on high-force interactions, while VidForce achieves the lowest overall mean peak MAE, tail-object error and profile RMSE among the evaluated priors.
For variation within a category, the routed prior provides an initial estimate that is adapted to the observed interaction. Instance adaptation reduces peak MAE from 6.9 to 4.8 N (Table 3), allowing interactions sharing a prior to receive different predictions. We have also added an analysis within each object–action group across execution speeds (Appendix C.14, Tables 44 and 45), where the error does not rise steadily with speed. Q1 gives the per-object numbers.
Our goal is to learn from limited sensory data. Our analysis shows that the prototypes provide an effort-based starting estimate, the instance adaptation adjusts its magnitude and shape to the observed interaction, and the joint flow refines the trajectory and corrects the remaining force error when the prior is weak (Tables 1 and 3). The stages therefore complement one another, reducing the sensitivity to the exact number of prototypes (Appendix C.8, Table 32) while supporting predictions across low-force, high-force and tail interactions (Table 4). Their combined benefit also shows in the full temporal force profile, where VidForce achieves the lowest RMSE among the evaluated priors (Table 4). We acknowledge data availability is limitation collecting more diverse data, curating one with better distribution fidelity and variance in physical properties in focus of our future efforts.