- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PACT estimates SMPL-X pose, per-point contact probabilities and 3D contact forces jointly from monocular video. It keeps SAM 3D Body frozen and adds learnable contact-force tokens (one per body-surface contact point), world-space kinematics from VGGT-Omega cameras, and a temporal transformer with iterative pose-offset refinement. Contact, force and gravity heads are trained on pseudo-labels from a physics-based annotation pipeline.
Critical strengths
-
Most video-to-force work is evaluated on foot–floor ground reaction forces. ForceWall measures hand and foot loads under changing supports, which greatly improve generalization.
-
The authors did comprehensive ablations and comparison experiments to show the effectiveness of the proposed method.
Critical weaknesses
-
The accuracy of the training labels has not been sufficiently validated. BEDLAM2 provides contact annotations only for the floor plane (Appendix A.7), meaning that hand-contact and hand-force supervision relies almost entirely on pseudo-labels from ClimbingVideos, whose accuracy remains uncertain.
-
The author claims that "end-to-end model beats staged models", but this claim requires stronger justification and experiments. The staged baselines differ from PACT in training data, body model, contact source and mass. For example, PhysPT never assigns force to the hands (Appendix C.6), so it cannot represent climbing loads.
Questions for authors
-
Please validate the annotation pipeline under the same configuration used for training. Specifically, run the ClimbingVideos pipeline on sequences with ground-truth annotations and report the accuracy of pose estimation, contact prediction, and force estimation.
-
The paper would benefit from including a better staged baseline trained on identical labels. For example, first estimate human motion using SAM3D Body and then train a small network to predict contacts and forces from the estimated motion. This would provide a more controlled comparison and help justify the claimed advantages of the end-to-end approach.
I am happy to increase my score if the author can address my concerns above.
4 comments
Response to Reviewer PEmX (2/2)
W2, Q2. The paper would benefit from including a better staged baseline trained on identical labels. For example, first estimate human motion using SAM3D Body and then train a small network to predict contacts and forces from the estimated motion. This would provide a more controlled comparison and help justify the claimed advantages of the end-to-end approach.
We thank the reviewer for the suggestion and evaluate the recommended baselines. Table C (Table 3 in the revised paper) compares PACT with three staged models, as described in more detail below. PACT has the lowest average force MAE of the four models, 87.5 N against 95.4 to 99.1 N for the staged ones (Table C, column Avg.), which supports the end-to-end approach.
All stages of the new baselines are trained on the same labels as PACT.
- (a) Staged, from motion. This is the baseline suggested by the reviewer. First, SAM 3D Body (Yang et al., 2026) estimates the human motion. Then a network predicts contacts and forces from this estimated motion only, without access to the image. The network has the same temporal transformer, heads and losses as PACT.
- (b) Staged, from motion and pose token. Same as (a), but the network also receives the pose token of SAM 3D Body. It still has no access to the image.
- (c) PhysPT, adapted. The reviewer notes that PhysPT (Zhang et al., 2024), one of the staged baselines of the paper, never assigns force to the hands. Training it on our labels would not change this, because its force model only produces force near a flat floor. We therefore replaced its force model by our 35 contact points, with a predicted contact and force for each point, and trained it from scratch on our labels.
Table C. Staged models trained on the same labels as PACT. Evaluation on ForceWall, all models trained on ClimbingVideos only. Columns as in Table 2 of the paper.
Given the results in Table C, PACT demonstrates the lowest force MAE on every limb (columns L. foot to R. hand) and the highest correlation with the sensors (0.729, column Corr). Its average force MAE is 87.5 N, against 95.8 N for (a) and 95.4 N for (b) (column Avg.). The main difference between PACT and (a) is that PACT has access to the image when it predicts contacts and forces. So the image contains information about contacts and forces that is lost when a model sees only the estimated motion. This is the advantage of the end-to-end approach.
The adapted PhysPT (c) has the best pose (44.5 mm, column MPJPE), since correcting the pose is its main objective. Its forces are the worst of the four models (99.1 N in column Avg., correlation 0.609).
We have added this experiment to Section 4.3 of the paper, and the three staged models are described in Appendix C.6.
We hope our new evidence addresses the concerns of the reviewer. We will be happy to provide further details if needed.
Follow-up on our response
Dear Reviewer PEmX,
Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address your two concerns, the accuracy of the training labels and the staged baseline trained on identical labels. If anything is still unclear, we would be glad to clarify it in the remaining time.
Best regards,
The authors
Many thanks to the authors for their rebuttal. Most of my concerns have been resolved, and I therefore increase my rating to 3.
My only remaining concern is that the human pose estimation error of the current annotation pipeline seems higher than expected. An MPJPE of 57.5 mm is relatively high, which raises concerns about whether ForceWall is sufficiently accurate to serve as a pose estimation benchmark. Moreover, the pose estimation results of PACT on ForceWall are not as good as those of PhysPT. I strongly encourage the authors to further investigate the reasons behind these issues and improve the annotation pipeline and PACT's pose estimation performance before releasing the code and data.
However, since the paper focuses primarily on contact and force estimation, I believe these limitations do not overshadow the overall value of the work. Therefore, I have decided to raise my score to 3.
Response to Reviewer PEmX (1/2)
We thank the reviewer for the clear summary and for naming the two claims that need more evidence. We address both below, with Table B for the label accuracy and a new controlled experiment in Table C for the staged comparison. The paper is revised accordingly.
W1, Q1. Please validate the annotation pipeline under the same configuration used for training. Specifically, run the ClimbingVideos pipeline on sequences with ground-truth annotations and report the accuracy of pose estimation, contact prediction, and force estimation.
Thanks for the suggestion. Please note that we did this validation in Appendix E of the original submission. There we run the annotation pipeline ("Optimization") on sequences with ground truth and report its pose, contact and force accuracy. In the revision we moved these results to Tables 1, 2 and 4 in the main paper. Table B below summarizes these results for the ForceWall and Parkour datasets. On the ClimbingVideos test set, the Optimization reaches a contact F1 of 0.892 against the manual annotation (Table 1).
Table B. Accuracy of the annotation pipeline (Optimization). Pose, contacts and forces on ForceWall and Parkour, next to the two baselines and PACT. Parkour has no contact ground truth, and the contacts of Li et al. (2019)++ are not scored.
MethodLi et al.++Inverse dynamics (InteractVLM contacts)Optimization (our labels)PACTMPJPE [mm]↓137.442.757.549.1ForceWallContacts F1↑−0.7410.9160.936Force MAE [N]↓141.4180.394.485.5Corr↑0.2430.0070.6540.726MPJPE [mm]↓95.368.765.766.3ParkourForce MAE [N]↓110.1170.6102.3102.4Corr↑0.5510.5270.6660.712
The Optimization (Table B, row 3) is more accurate than the two baselines on contacts and forces. On ForceWall its force MAE is 94.4 N against 141.4 N for Li et al.++ (row 1) and 180.3 N for inverse dynamics (row 2). Its correlation with the sensors is 0.654 against 0.243 and 0.007, and its contact F1 is 0.916 against 0.741 for inverse dynamics. However, the Optimization takes about 22 minutes for a 300-frame clip and fails on hard in-the-wild videos (Figure 4). PACT, trained on pseudo-labels obtained from the Optimization, requires only 4 seconds for the forward pass on the same clip and outperforms the Optimization on ForceWall (row 4, 85.5 N and correlation 0.726), thanks to its generalization capabilities.
W1, second part. BEDLAM2 provides contact annotations only for the floor plane (Appendix A.7), meaning that hand-contact and hand-force supervision relies almost entirely on pseudo-labels from ClimbingVideos, whose accuracy remains uncertain
The reviewer is right that hand supervision comes mostly from the ClimbingVideos pseudo-labels. BEDLAM2 (Tesch et al., 2025) only adds a few hand contacts with the floor. However, please note that we evaluate the hands on ForceWall, where the hand forces are measured by sensors. There the accuracy of the pseudo-labels is good. In Table 2 (ForceWall block, columns L. hand and R. hand), the force MAE of the Optimization on the left and right hand is 89.2 and 84.9 N, against 128.8 and 129.1 N for the best baseline, Li et al. (2019)++. PACT, trained on these pseudo-labels, reaches 72.9 and 80.4 N. Hence, when evaluating PACT on real force sensor ground truth, it obtains more accurate estimates of hand forces compared to the baselines and the pseudo-labels themselves.