acceptodds
Under review as a conference paper at ICLR 2027

PACT: End-to-End Learning of Human Pose, Contacts, and Forces from Video

Abstract

Human motion, environmental contacts, and interaction forces are governed by common physical laws, yet existing approaches typically separate visual pose reconstruction from contact and force estimation. This separation limits joint reasoning and can propagate errors between stages. We introduce PACT, an end-to-end model that jointly learns to estimate human pose, contacts and contact forces from monocular video. Our approach augments a human reconstruction foundation model with learnable contact–force tokens and a temporal transformer that integrates visual features with world-space motion. Joint prediction heads refine human poses and estimate contacts and forces, while physics-based supervision encourages consistency between the reconstructed motion and interaction forces. To address the scarcity of force annotations, we develop a data annotation pipeline that combines contact labeling with physics-based motion and force optimization, producing training supervision from synthetic and real-world videos. We also introduce a real-world climbing benchmark ForceWall with climbing videos and corresponding ground truth contact forces obtained from the force sensors. Experiments demonstrate state-of-the-art contact and force estimation, outperforming staged reconstruction approaches and generalizing to interactions beyond the training distribution. These results support end-to-end joint learning as an effective approach to recovering human motion and physical interactions from video.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
contact force estimation, contact estimation, 3D human pose estimation, physics-based human motion, monocular video

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_PEmXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACT estimates SMPL-X pose, per-point contact probabilities and 3D contact forces jointly from monocular video. It keeps SAM 3D Body frozen and adds learnable contact-force tokens (one per body-surface contact point), world-space kinematics from VGGT-Omega cameras, and a temporal transformer with iterative pose-offset refinement. Contact, force and gravity heads are trained on pseudo-labels from a physics-based annotation pipeline.

Critical strengths
  1. Most video-to-force work is evaluated on foot–floor ground reaction forces. ForceWall measures hand and foot loads under changing supports, which greatly improve generalization.

  2. The authors did comprehensive ablations and comparison experiments to show the effectiveness of the proposed method.

Critical weaknesses
  1. The accuracy of the training labels has not been sufficiently validated. BEDLAM2 provides contact annotations only for the floor plane (Appendix A.7), meaning that hand-contact and hand-force supervision relies almost entirely on pseudo-labels from ClimbingVideos, whose accuracy remains uncertain.

  2. The author claims that "end-to-end model beats staged models", but this claim requires stronger justification and experiments. The staged baselines differ from PACT in training data, body model, contact source and mass. For example, PhysPT never assigns force to the hands (Appendix C.6), so it cannot represent climbing loads.

Questions for authors
  1. Please validate the annotation pipeline under the same configuration used for training. Specifically, run the ClimbingVideos pipeline on sequences with ground-truth annotations and report the accuracy of pose estimation, contact prediction, and force estimation.

  2. The paper would benefit from including a better staged baseline trained on identical labels. For example, first estimate human motion using SAM3D Body and then train a small network to predict contacts and forces from the estimated motion. This would provide a more controlled comparison and help justify the claimed advantages of the end-to-end approach.

I am happy to increase my score if the author can address my concerns above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer PEmX (1/2)

We thank the reviewer for the clear summary and for naming the two claims that need more evidence. We address both below, with Table B for the label accuracy and a new controlled experiment in Table C for the staged comparison. The paper is revised accordingly.

W1, Q1. Please validate the annotation pipeline under the same configuration used for training. Specifically, run the ClimbingVideos pipeline on sequences with ground-truth annotations and report the accuracy of pose estimation, contact prediction, and force estimation.

Thanks for the suggestion. Please note that we did this validation in Appendix E of the original submission. There we run the annotation pipeline ("Optimization") on sequences with ground truth and report its pose, contact and force accuracy. In the revision we moved these results to Tables 1, 2 and 4 in the main paper. Table B below summarizes these results for the ForceWall and Parkour datasets. On the ClimbingVideos test set, the Optimization reaches a contact F1 of 0.892 against the manual annotation (Table 1).

Table B. Accuracy of the annotation pipeline (Optimization). Pose, contacts and forces on ForceWall and Parkour, next to the two baselines and PACT. Parkour has no contact ground truth, and the contacts of Li et al. (2019)++ are not scored.

ForceWallParkourMethodMPJPE [mm]↓Contacts F1↑Force MAE [N]↓Corr↑MPJPE [mm]↓Force MAE [N]↓Corr↑Li et al.++137.4−141.40.24395.3110.10.551Inverse dynamics (InteractVLM contacts)42.70.741180.30.00768.7170.60.527Optimization (our labels)57.50.916‾94.4‾0.654‾65.7102.30.666‾PACT49.1‾0.93685.50.72666.3‾102.4‾0.712\small\begin{array}{l|cccc|ccc}\hline&&\clap{\kern7.0em\textbf{ForceWall}}&&&&\clap{\textbf{Parkour}}&\\\\\textbf{Method}&\text{MPJPE [mm]↓}&\text{Contacts F1↑}&\text{Force MAE [N]↓}&\text{Corr↑}&\text{MPJPE [mm]↓}&\text{Force MAE [N]↓}&\text{Corr↑}\\\\\hline\text{Li et al.++}&137.4&-&141.4&0.243&95.3&110.1&0.551\\\\\text{Inverse dynamics (InteractVLM contacts)}&\mathbf{42.7}&0.741&180.3&0.007&68.7&170.6&0.527\\\\\text{Optimization (our labels)}&57.5&\underline{0.916}&\underline{94.4}&\underline{0.654}&\mathbf{65.7}&\mathbf{102.3}&\underline{0.666}\\\\\hline\text{PACT}&\underline{49.1}&\mathbf{0.936}&\mathbf{85.5}&\mathbf{0.726}&\underline{66.3}&\underline{102.4}&\mathbf{0.712}\\\\\hline\end{array}

The Optimization (Table B, row 3) is more accurate than the two baselines on contacts and forces. On ForceWall its force MAE is 94.4 N against 141.4 N for Li et al.++ (row 1) and 180.3 N for inverse dynamics (row 2). Its correlation with the sensors is 0.654 against 0.243 and 0.007, and its contact F1 is 0.916 against 0.741 for inverse dynamics. However, the Optimization takes about 22 minutes for a 300-frame clip and fails on hard in-the-wild videos (Figure 4). PACT, trained on pseudo-labels obtained from the Optimization, requires only 4 seconds for the forward pass on the same clip and outperforms the Optimization on ForceWall (row 4, 85.5 N and correlation 0.726), thanks to its generalization capabilities.

W1, second part. BEDLAM2 provides contact annotations only for the floor plane (Appendix A.7), meaning that hand-contact and hand-force supervision relies almost entirely on pseudo-labels from ClimbingVideos, whose accuracy remains uncertain

The reviewer is right that hand supervision comes mostly from the ClimbingVideos pseudo-labels. BEDLAM2 (Tesch et al., 2025) only adds a few hand contacts with the floor. However, please note that we evaluate the hands on ForceWall, where the hand forces are measured by sensors. There the accuracy of the pseudo-labels is good. In Table 2 (ForceWall block, columns L. hand and R. hand), the force MAE of the Optimization on the left and right hand is 89.2 and 84.9 N, against 128.8 and 129.1 N for the best baseline, Li et al. (2019)++. PACT, trained on these pseudo-labels, reaches 72.9 and 80.4 N. Hence, when evaluating PACT on real force sensor ground truth, it obtains more accurate estimates of hand forces compared to the baselines and the pseudo-labels themselves.

AuthorsOpenReview ↗

Response to Reviewer PEmX (2/2)

W2, Q2. The paper would benefit from including a better staged baseline trained on identical labels. For example, first estimate human motion using SAM3D Body and then train a small network to predict contacts and forces from the estimated motion. This would provide a more controlled comparison and help justify the claimed advantages of the end-to-end approach.

We thank the reviewer for the suggestion and evaluate the recommended baselines. Table C (Table 3 in the revised paper) compares PACT with three staged models, as described in more detail below. PACT has the lowest average force MAE of the four models, 87.5 N against 95.4 to 99.1 N for the staged ones (Table C, column Avg.), which supports the end-to-end approach.

All stages of the new baselines are trained on the same labels as PACT.

  • (a) Staged, from motion. This is the baseline suggested by the reviewer. First, SAM 3D Body (Yang et al., 2026) estimates the human motion. Then a network predicts contacts and forces from this estimated motion only, without access to the image. The network has the same temporal transformer, heads and losses as PACT.
  • (b) Staged, from motion and pose token. Same as (a), but the network also receives the pose token of SAM 3D Body. It still has no access to the image.
  • (c) PhysPT, adapted. The reviewer notes that PhysPT (Zhang et al., 2024), one of the staged baselines of the paper, never assigns force to the hands. Training it on our labels would not change this, because its force model only produces force near a flat floor. We therefore replaced its force model by our 35 contact points, with a predicted contact and force for each point, and trained it from scratch on our labels.

Table C. Staged models trained on the same labels as PACT. Evaluation on ForceWall, all models trained on ClimbingVideos only. Columns as in Table 2 of the paper.

Force MAE [N]↓Force agreementResidual↓ModelMPJPE [mm]↓L. footR. footL. handR. handAvg.Angle [°]↓Corr↑Share [pp]↓Force [bw]Moment [bw m](a) Staged, from motion63.6119.6‾104.378.980.3‾95.816.40.667‾9.2‾0.1840.039(b) Staged, from motion and pose token61.5124.299.4‾77.0‾80.995.4‾16.10.6639.40.173‾0.036(c) PhysPT, adapted44.5119.6‾116.578.581.899.114.40.60911.60.2530.050PACT (end-to-end)56.6‾106.294.872.276.887.515.0‾0.7298.60.1550.037‾\small\begin{array}{lc|ccccc|ccc|cc}\hline&&&&\clap{\textbf{Force MAE [N]↓}}&&&&\clap{\textbf{Force agreement}}&&\clap{\kern6.0em\textbf{Residual↓}}&\\\\\textbf{Model}&\text{MPJPE [mm]↓}&\text{L. foot}&\text{R. foot}&\text{L. hand}&\text{R. hand}&\text{Avg.}&\text{Angle [°]↓}&\text{Corr↑}&\text{Share [pp]↓}&\text{Force [bw]}&\text{Moment [bw m]}\\\\\hline\text{(a) Staged, from motion}&63.6&\underline{119.6}&104.3&78.9&\underline{80.3}&95.8&16.4&\underline{0.667}&\underline{9.2}&0.184&0.039\\\\\text{(b) Staged, from motion and pose token}&61.5&124.2&\underline{99.4}&\underline{77.0}&80.9&\underline{95.4}&16.1&0.663&9.4&\underline{0.173}&\mathbf{0.036}\\\\\text{(c) PhysPT, adapted}&\mathbf{44.5}&\underline{119.6}&116.5&78.5&81.8&99.1&\mathbf{14.4}&0.609&11.6&0.253&0.050\\\\\hline\text{PACT (end-to-end)}&\underline{56.6}&\mathbf{106.2}&\mathbf{94.8}&\mathbf{72.2}&\mathbf{76.8}&\mathbf{87.5}&\underline{15.0}&\mathbf{0.729}&\mathbf{8.6}&\mathbf{0.155}&\underline{0.037}\\\\\hline\end{array}

Given the results in Table C, PACT demonstrates the lowest force MAE on every limb (columns L. foot to R. hand) and the highest correlation with the sensors (0.729, column Corr). Its average force MAE is 87.5 N, against 95.8 N for (a) and 95.4 N for (b) (column Avg.). The main difference between PACT and (a) is that PACT has access to the image when it predicts contacts and forces. So the image contains information about contacts and forces that is lost when a model sees only the estimated motion. This is the advantage of the end-to-end approach.

The adapted PhysPT (c) has the best pose (44.5 mm, column MPJPE), since correcting the pose is its main objective. Its forces are the worst of the four models (99.1 N in column Avg., correlation 0.609).

We have added this experiment to Section 4.3 of the paper, and the three staged models are described in Appendix C.6.

We hope our new evidence addresses the concerns of the reviewer. We will be happy to provide further details if needed.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer PEmX,

Thank you again for your review. As the discussion period is ending soon, we wanted to check whether our response and the revised paper address your two concerns, the accuracy of the training labels and the staged baseline trained on identical labels. If anything is still unclear, we would be glad to clarify it in the remaining time.

Best regards,

The authors

Many thanks to the authors for their rebuttal. Most of my concerns have been resolved, and I therefore increase my rating to 3.

My only remaining concern is that the human pose estimation error of the current annotation pipeline seems higher than expected. An MPJPE of 57.5 mm is relatively high, which raises concerns about whether ForceWall is sufficiently accurate to serve as a pose estimation benchmark. Moreover, the pose estimation results of PACT on ForceWall are not as good as those of PhysPT. I strongly encourage the authors to further investigate the reasons behind these issues and improve the annotation pipeline and PACT's pose estimation performance before releasing the code and data.

However, since the paper focuses primarily on contact and force estimation, I believe these limitations do not overshadow the overall value of the work. Therefore, I have decided to raise my score to 3.

Reviewer_2wGyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PACT, an end-to-end approach to simultaneously predict human pose, contacts and forces from videos. The method is built on top of SAM 3D Body and VGGT-Ω\Omega, with additional Transformer and regression heads to predict the respective dynamics quantities. The (pseudo) ground truths are obtained via an optimization process based on Newtonian motion mechanics and scene point clouds obtained from VGGT-Ω\Omega. The model PACT is trained on BEDLAM2 and a new dataset containing climbing videos.

Critical strengths
  • One contribution of the paper is the leverage of recent advancements such as VGGT-Ω\Omega, SAM 3D Body, and Sapiens2 to generate high-quality pseudo-GT contact and force data for training. The training on a massive database like BEDLAM2 also requires substantial computing resources.
  • The proposed PACT achieves best performance on the two climbing datasets and competitive on the RICH dataset.
  • The components of PACT are individually verified, with the trade-offs between pose accuracy and force estimation plausibility.
Critical weaknesses
  • It is unclear how the physics-driven kinematics qq is obtained to realize the kinematic objective Ekin(q)E_{kin}(q) in Eq. 8. In #267, a simulation is referred to, but I could not find any relevant information.
  • Also, in Eq. 8, I could not see how else the variable ff contributes towards the minimization of the objective but becoming zero. The second and third term seem to only regularize the magnitude of the force, with τtf\tau_{t}^{f} and τtm\tau_{t}^{m} are propagated from ff via the Jacobian matrix (if floating-base).
  • Given the comparison in Fig. 3, PACT predicts seem-to-be more plausible forces than the optimization method. This makes me skeptical about the integrity of the evaluations on the CLIMBINGVIDEOS dataset in Tab. 1.
  • To be honest, the content of the paper is too long (>16 pages of appendix), and many of the arguments made in the paper are backed by separate sections in the supplementary, making it not self-contained. It is impossible to produce detailed reviews within the timeline of the conference.
Questions for authors

Q1: Is it easy for PACT to extend the number of contact points to more than K=35K=35? Would it require full re-training of the method?

Q2: How was the velocity q˙\dot{q} and acceleration q¨\ddot{q} are obtained? An SMPL motion estimated from videos are often very noisy (in global translations and rotations), due to the ill-pose recovery from 2D to 3D. This could make the corresponding velocity and acceleration extremely high.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2wGy (1/2)

We thank the reviewer for the careful reading of the method. We believe the first two concerns come from sentences in Section 3.2 that were not clear enough. We clarify them below and in the revised text.

W1. It is unclear how the physics-driven kinematics q\mathbf{q} is obtained to realize the kinematic objective Ekin(q)E_{\text{kin}}(\mathbf{q}) in Eq. 8. In #267, a simulation is referred to, but I could not find any relevant information.

We do not use simulation anywhere in the PACT pipeline. Instead we rely on optimization-based inverse dynamics for producing pseudo-labels, L267 only mentions simulation for contrasting purposes. In a simulation the scene is known and contacts and forces come for free, while from video they have to be estimated. We reformulated that sentence in the revision (Section 3.2, L267--268).

The kinematics q\mathbf{q} are obtained by optimization. We start from the SAM 3D Body (Yang et al., 2026) pose and minimize Eq. 8 over q\mathbf{q} and the forces f\mathbf{f}. The term Ekin(q)E_{\text{kin}}(\mathbf{q}) keeps the body close to the video. It penalizes the distance to the 2D keypoints, to the initial pose and to the reconstructed scene, together with pose priors and temporal smoothness (Appendix A.3). The other terms of Eq. 8 are the physics, and they make this motion consistent with the forces.

W2. In Eq. 8, I could not see how else the variable f\mathbf{f} contributes towards the minimization of the objective but becoming zero. The second and third term seem to only regularize the magnitude.

We agree that Eq. 8 does not show this clearly, since f\mathbf{f} does not appear in the second and third terms. These two terms are not regularizers of the magnitude of f\mathbf{f}. They are the residual force τf\boldsymbol{\tau}^f and the residual moment τm\boldsymbol{\tau}^m, and both depend on f\mathbf{f} through Eq. 7

τt=M(qt),q¨t+b(qt,q˙t)+G(qt)⏟force that the motion needs−J(qt)⊤ft⏟force that the contacts give,\boldsymbol{\tau}_t = \underbrace{\mathbf{M}(\mathbf{q}_t)\\,\ddot{\mathbf{q}}_t + \mathbf{b}(\mathbf{q}_t,\dot{\mathbf{q}}_t) + \mathbf{G}(\mathbf{q}_t)}_{\text{force that the motion needs}} - \underbrace{\mathbf{J}(\mathbf{q}_t)^{\top}\mathbf{f}_t}_{\text{force that the contacts give}} ,

where τf\boldsymbol{\tau}^f and τm\boldsymbol{\tau}^m are the first six entries of τt\boldsymbol{\tau}_t. So the residual is the difference between the force that the motion needs and the force that the contacts give. It is zero when the two are equal, not when f\mathbf{f} is zero.

For example, take a person standing still. Gravity pulls the body down, so the ground has to push up with one body weight. If f=0\mathbf{f} = 0, nothing holds the person, and the residual is the whole body weight, the largest it can be. If f\mathbf{f} is one body weight pointing up, the residual is zero. Minimizing the second and third terms therefore moves f\mathbf{f} towards the forces that explain the motion, and away from zero.

Minimizing this residual is a common way to estimate contact forces from motion, used for example by Li et al. (2019). In the revision we state right after Eq. 8 that τf\boldsymbol{\tau}^f and τm\boldsymbol{\tau}^m depend on f\mathbf{f} and are zero only when the contact forces explain the motion (L287--291).

W3. Given the comparison in Fig. 3, PACT predicts seem-to-be more plausible forces than the optimization method. This makes me skeptical about the integrity of the evaluations on the ClimbingVideos dataset in Tab. 1.

The Optimization can fail as in Fig. 3 and in the new Figure 4, but its contacts are not used as ground truth in Table 1. The Optimization fits each video separately, so when the estimated pose or scene of that video is wrong, the contacts and forces are wrong as well. PACT is trained on many videos, and we believe it is less sensitive to such errors because they differ from video to video. The Optimization does estimate contacts, but we use them only as training labels. For the ClimbingVideos test set the contacts were annotated manually. On every frame of every test video, an annotator marked which of 14 body joints are in contact (Appendix C.1, Figure 8), and the test videos are different from the training videos. In Table 1 the Optimization is itself scored against these manual annotations, like any other method.

The other two datasets of Table 1 are also independent of the Optimization. On ForceWall the contacts are measured by the force sensors, and RICH (Huang et al., 2022) comes with its own annotations. The failures of the Optimization therefore affect the training labels and not the evaluation in Table 1.

AuthorsOpenReview ↗

Response to Reviewer 2wGy (2/2)

W4. The content of the paper is too long (>16 pages of appendix), and many of the arguments made in the paper are backed by separate sections in the supplementary, making it not self-contained.

We understand the concern. The appendix is long because we chose to document every constant of the pipeline and of the training, so that the labels and the results can be reproduced. The revision makes the main paper more self-contained. We made the following changes:

  • The evaluation of the Optimization moved from the appendix into Tables 1, 2 and 4 (L391, L403, L408, L443).
  • Section 3.2 now describes EkinE_{\text{kin}} and the role of f\mathbf{f} in Eq. 8 (L287--291).
  • Section 3.2 now states how velocities and accelerations are computed (L315--316).

With these changes the key concepts and the main results are in the main paper, and the appendix contains only implementation details and additional validations.

Q1. Is it easy for PACT to extend the number of contact points to more than K=35K=35? Would it require full re-training of the method?

Increasing the number of contact points is easy and does not require full retraining. We chose K=35K=35 for simplicity. These points cover the hands, the feet and the other body parts that touch the scene in the motions we study. To add a point, PACT only needs one new input embedding, since the contact and force heads are shared between all points. The new embedding can be trained while the rest of the model stays frozen. Doubling KK to 70 adds about 2 percent to the time of the forward pass. While the computational cost is small, more points require training labels for these new points, which we leave for future work. We added this discussion to Appendix D.

Q2. How was the velocity q˙\dot{q} and acceleration q¨\ddot{q} are obtained? An SMPL motion estimated from videos are often very noisy (in global translations and rotations), due to the ill-pose recovery from 2D to 3D. This could make the corresponding velocity and acceleration extremely high.

We agree that the motion estimated per frame is too noisy for this. In both cases q˙\dot{\mathbf{q}} and q¨\ddot{\mathbf{q}} are finite differences in time of a smooth trajectory, and never of the raw per-frame estimate.

For the Optimization, the trajectory q\mathbf{q} is itself optimized. Its objective penalizes the second, third and fourth time derivatives of the pose (Appendix A.3, Eq. 17), so the optimized trajectory has no sudden changes in acceleration. q˙\dot{\mathbf{q}} and q¨\ddot{\mathbf{q}} are the central differences of this trajectory (Appendix A.4).

For PACT, a temporal transformer takes the per-frame poses of a whole window and predicts a corrected trajectory. It is trained with a loss on joint velocities in addition to positions (Appendix B.6, Eq. 25), so its output is smooth. q˙\dot{\mathbf{q}} and q¨\ddot{\mathbf{q}} in the physics loss are the central differences of this corrected trajectory (Appendix B.7).

On ForceWall the raw SAM 3D Body (Yang et al., 2026) motion has about seven times more joint jitter than our outputs. We added one sentence for each case to Section 3.

Reviewer_2wGyOpenReview ↗

Response to authors

Many thanks to the authors for the rebuttal. All of my concerns have been resolved. I thereby increase the rating to 4: Clear acceptance.

Reviewer_gZMvOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduced PACT, an end-to-end model to estimate human dynamics from monocular video. This largely involves extending SAM 3D Body (3DB) to further predict extrinsics, contact states, and contact forces, appending learnable contact-force tokens to 3DB's decoder query sequence, and incorporating dynamics and contact losses during training. The authors also introduced an annotation pipeline that combines kinematic detections from SAM3, Sapiens2, 3DB, VGGT-Ω\Omega for inverse dynamics optimization to obtain contact-force labels for training. Lastly, the authors collected a climbing dataset, ForceWall, with force sensors attached to a climbing wall, synchronized with video cameras.

Critical strengths

This is a good paper. It is well written with clear motivations, contributions, and methodology, and includes extensive experiments. Furthermore, the collected ForceWall dataset provides useful contact and force labels for human dynamics reconstruction research.

Critical weaknesses

While PACT's overall force error still seems high (e.g. 85.5N per limb MAE for ForceWall), so is the case for competing methods, and PACT outperforms them in most cases. However, I'm more interested with the worse results from the annotation pipeline's optimization method (e.g. 94.5N), and potential learning implications. Specifics listed in questions section.

Questions for authors

(the annotation pipeline's optimization method will be referred to as "Optimization" here, as in Tabs 11-13)

  1. I'm guessing the worse results for Optimization can be attributed to the accumulation of kinematic detection errors. For example, the training data annotation in 3DB is semi-manual, as they were reinforced with human adjustments, whereas it seems fully automated here. During the contact annotation for the ClimbingVideos test split, were 2D keypoint detections also manually updated?

  2. In Page 8 Line 430, the paper claims that PACT generalizes beyond the capability of its own annotation pipeline. I'm wondering if PACT is actually learning better force relations than Optimization, or instead learning to correct Optimization's potential systematic kinematic estimation errors, and hence leading to better average force results, while preserving the force relations from Optimization.
    In Tabs 12-13, adding a comparison between PACT's force prediction and those from Optimization using PACT's kinematics prediction as input, could provide some insight into this.

  3. Furthermore, in Tab 12, for Parkour, when Optimization's kinematics estimates are actually slightly better than PACT's, Optimization's average force MAE is also slightly better, while having significantly lower residual force. Why do you think this is the case? Also, the error distribution between the left and right foot seem more balanced for Optimization than PACT.

  4. The previous claim (P8 L430) is followed with examples in Fig 3 showing cases where PACT produced better force estimations than Optimization. Supplementing these cases with comparisons in kinematics predictions would be helpful. For example, in the left top figure of Fig 3, the Optimization's force estimate is absent. Would this be because Optimization's output pose is not in contact with the beam?

  5. Overall, the Optimization's force errors still seem high and I was surprised to see that it was worse than both OpenCap methods, despite the OpenCap methods having zero residual force. How do the kinematic errors compare in Tab 3 and Tab 13?

  6. How do the output forces compare when the ground truth OpenCap kinematics is fed into Optimization's inverse-dynamics solver? Some baseline to evaluate the inverse dynamics part of the annotation pipeline would be helpful.

  7. Moving some of Optimization's appendix results (e.g. Tabs 11-13) into the main paper (e.g. Tabs 2-3) would be helpful.

  8. What is the inference speed like for the Annotation pipeline?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gZMv (1/3)

We thank the reviewer for the positive assessment and the thoughtful questions. They led to a new experiment, Table A below (Table 12 in Appendix E.3 of the revised paper). Our response has three parts, Q1 and Q2 here, Q3 and Q4 in part 2/3 and Q5 to Q8 in part 3/3.

As in the review, "Optimization" is our annotation pipeline of Section 3.2, which produces the training labels of PACT. It takes the human pose estimated from the video as input, refines it, and estimates contacts and forces by optimization. In the tables it is evaluated on the test sets like any other method.

Q1. During the contact annotation for the ClimbingVideos test split, were 2D keypoint detections also manually updated?

No, the contact annotation specifies joint IDs and their in-contact time intervals, it does not require precise localization of the joints. Annotators watched the videos and marked each of the 14 joints as in-contact, free or unknown (Figure 8). The Sapiens2 keypoints are only drawn on the video to help locate a joint quickly. A keypoint a few pixels off does not change whether a joint is seen as touching the wall. The evaluation uses only the contact state of each joint, not its location, so the keypoints do not enter the metric. We now clarify this in Appendix C.1, thank you for the question.

Q2. I'm wondering if PACT is actually learning better force relations than Optimization, or instead learning to correct Optimization's potential systematic kinematic estimation errors, and hence leading to better average force results, while preserving the force relations from Optimization.

To clarify this question, we compare PACT with Optimization initialized using human poses obtained from: 1. SAM 3D Body; 2. PACT; 3. Ground truth, see Table A. Row 2 answers this question. There the Optimization starts from the pose of PACT, so the two have the same pose error and differ only in how they estimate forces.

With the pose of PACT, the force MAE of the Optimization on ForceWall drops from 94.4 N (row 1, column All) to 85.9 N (row 2, column All), about the same as PACT with 85.5 N (row 4, column All). The trend is the same on Parkour. So we agree that a large part of PACT's gain over the Optimization comes from correcting its kinematic errors.

Still, we believe PACT also learns better force relations. With the same pose, PACT has a higher correlation with the ForceWall sensors than the Optimization (0.726 vs 0.691, rows 4 and 2, column Corr) and a lower force error on OpenCap walking (11.8 vs 16.2 percent of body weight, column Walking). In addition, PACT trained on ClimbingVideos only has about the same pose error as the Optimization of row 1, but a lower force MAE (87.5 vs 94.4 N, Table 3 of the revised paper, last row). A possible reason is that the Optimization sees only keypoints, while PACT sees the image. Finally, PACT needs only one forward pass, with no optimization at test time.

ForceWallParkourOpenCapInput posePose↓F1↑Hands↓Feet↓All↓Corr↑Pose↓All↓Corr↑Walking↓All↓1. SAM 3D Body57.50.91687.1101.894.40.65465.7102.30.66619.213.62. PACT48.90.91875.896.085.90.69166.396.20.71216.215.53. Ground truth−0.94865.278.972.00.807−−−14.39.74. PACT (no optimization)49.10.93676.794.285.50.72666.3102.40.71211.811.6\begin{array}{l|cccccc|ccc|cc}\hline&&&\clap{\kern4.0em\textbf{ForceWall}}&&&&&\clap{\textbf{Parkour}}&&\clap{\kern5.0em\textbf{OpenCap}}&\\\\\textbf{Input pose}&\text{Pose↓}&\text{F1↑}&\text{Hands↓}&\text{Feet↓}&\text{All↓}&\text{Corr↑}&\text{Pose↓}&\text{All↓}&\text{Corr↑}&\text{Walking↓}&\text{All↓}\\\\\hline\text{1. SAM 3D Body}&57.5&0.916&87.1&101.8&94.4&0.654&65.7&102.3&0.666&19.2&13.6\\\\\text{2. PACT}&48.9&0.918&75.8&96.0&85.9&0.691&66.3&96.2&0.712&16.2&15.5\\\\\text{3. Ground truth}&-&0.948&65.2&78.9&72.0&0.807&-&-&-&14.3&9.7\\\\\hline\text{4. }\textbf{PACT}\text{ (no optimization)}&49.1&0.936&76.7&94.2&85.5&0.726&66.3&102.4&0.712&11.8&11.6\\\\\hline\end{array} Table A. The Optimization with different input poses. Rows 1 to 3 are the Optimization, with the input pose from SAM 3D Body as for our training labels (row 1), from PACT (row 2) or from the ground truth (row 3). Row 4 is PACT itself, for comparison. Pose is MPJPE in mm, F1 the contact F1, Hands, Feet and All the force MAE in N, Corr the correlation with the sensors, OpenCap the vertical force error in percent of body weight.

AuthorsOpenReview ↗

Response to Reviewer gZMv (2/3)

Q3. For Parkour, when Optimization's kinematics estimates are actually slightly better than PACT's, Optimization's average force MAE is also slightly better, while having significantly lower residual force. Why do you think this is the case? Also, the error distribution between the left and right foot seem more balanced for Optimization than PACT

Indeed, on Parkour the Optimization is slightly ahead in kinematics and average force MAE, although the two are very close (102.3 vs 102.4 N, Table 2, Parkour block, column Avg.) and PACT has the higher correlation with the sensors (0.712 vs 0.666, column Corr). A likely reason is that Parkour-like motions are not part of PACT's training data, while the Optimization fits each test video and needs no training data.

The lower residual force of the Optimization is expected, since it minimizes this residual on each test video, while PACT has no optimization at test time. A lower residual means that the forces agree better with the estimated motion, not that they are closer to the measured forces. Inverse dynamics, for example, has the lowest residual force in Table 2 and the highest force MAE.

The imbalance between the feet comes from the right foot alone, where PACT has 148.4 N against 124.8 N for the Optimization (Table 2, Parkour block, column R. foot). On the other three limbs PACT is better or about equal. We believe the reason is that the Parkour camera often sees only one side of the body. During training we exclude non-visible limbs from the contact and force losses, because their labels are uncertain, so PACT learns less about hidden limbs. The Optimization fits each test video and treats both sides alike.

Q4. The previous claim (P8 L430) is followed with examples in Fig 3 showing cases where PACT produced better force estimations than Optimization. Supplementing these cases with comparisons in kinematics predictions would be helpful. For example, in the left top figure of Fig 3, the Optimization's force estimate is absent. Would this be because Optimization's output pose is not in contact with the beam?

Yes, this is exactly the reason. The new Figure 4 of the revised paper shows the pose, contacts and reconstructed 3D scene of the Optimization next to the pose and contacts of PACT, for the four examples of Figure 3 (balance beam, pommel horse, aerial dance and climbing dyno). In the top-left example of Figure 3, the balance beam, the body of the Optimization floats about 35 cm above the beam when the athlete pushes off the beam before the jump, so no point on the body is close enough to the beam to be labeled as a contact and the solver assigns no force, while PACT labels the right foot. In the other three examples the Optimization fails for a similar reason, it gets the scene or the contacts wrong before it estimates any force. In the pommel horse example the 3D scene reconstructed from the video puts the arena background at the same depth as the athlete. In the aerial dance example the reconstructed scene has no floor. In the climbing dyno example the back, shoulders, hips and feet are wrongly labeled as touching the wall. Wrong contacts then give wrong forces. PACT does not use the reconstructed scene and predicts contacts and forces directly from the image.

AuthorsOpenReview ↗

Response to Reviewer gZMv (3/3)

Q5. The Optimization's force errors still seem high and I was surprised to see that it was worse than both OpenCap methods, despite the OpenCap methods having zero residual force. How do the kinematic errors compare in Tab 3 and Tab 13?

The OpenCap methods (Uhlrich et al., 2023; Gilon et al., 2026) estimate forces with a physics simulator. They reproduce the motion in a musculoskeletal simulation with a foot-ground contact model for a flat floor, so the equations of motion hold exactly. This is why their residual force is zero, and it is also why their ground forces are accurate on walking. The Optimization uses no simulator and no flat-floor assumption, since it must also work for sports such as climbing. Physics is only a soft term in its objective, which is less accurate on walking.

To compare the kinematic errors, we fitted an SMPL-X body to the OpenCap marker data, following MoSh++, and computed the MPJPE of each method against it over all 64 OpenCap trials. This comparison is new and is given in the table below.

SAM 3D BodyOptimizationPACT
MPJPE [mm]↓59.857.445.2

PACT has the lowest kinematic error. The Optimization is only slightly better than the SAM 3D Body pose it starts from. PACT is better because it is also trained on BEDLAM2, which has ground-truth poses of many walking people.

The kinematic error explains only a part of the force error of the Optimization. With ground-truth kinematics as input, the force error of the Optimization on OpenCap walking drops from 19.2 to 14.3 percent of body weight (Table A in part 1/3, rows 1 and 3, column Walking), and PACT is still better with 11.8 (row 4). So the remaining force error of the Optimization comes mostly from how it estimates contacts and forces, and not from its kinematics.

Q6. How do the output forces compare when the ground truth OpenCap kinematics is fed into Optimization's inverse-dynamics solver?

As suggested, we also replace the estimated kinematics with the ground truth ones, establishing a baseline for evaluating the inverse dynamics contribution (Table A in part 1/3, row 3, compared with row 1 where the kinematics are estimated). Alongside OpenCap, we also report ground truth kinematics on ForceWall, estimated from the multi-view reference motion (Appendix C.2). On ForceWall, the force MAE drops from 94.4 to 72.0 N (rows 1 and 3, column All) and the correlation goes up from 0.654 to 0.807 (column Corr). On OpenCap walking, the error drops from 19.2 to 14.3 percent of body weight (column Walking). This means that even with ground truth motion, a significant part of the error remains, and it is to be attributed to inverse dynamics. We believe that this is also due to motion ambiguity, as the split of the load between limbs is ambiguous for any inverse dynamics solver, we leave this issue for future work.

Q7. Moving some of Optimization's appendix results (e.g. Tabs 11-13) into the main paper (e.g. Tabs 2-3) would be helpful.

Done. The Optimization rows are now in Tables 1, 2 and 4 of the main paper. In the appendix, the old comparison tables are replaced by Table 12 (Appendix E.3), which is Table A of part 1/3.

Q8. What is the inference speed like for the Annotation pipeline?

On one Nvidia RTX Pro6000 a 300-frame 1080p clip takes about 22 minutes. Three quarters of this time is the inference of the foundation models (the Sapiens2 keypoints alone take about half), and the optimization passes take about four minutes. The forward pass of PACT takes about 4 seconds for the same clip, of which the temporal refiner takes 40 ms (tracking and camera excluded). The runtime is now reported in Section 4.3.

We hope this answers the questions, and we are happy to discuss further.

I really appreciate the detailed responses. My score maintains. Great job!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers for their time and constructive feedback. Below we summarize the main concerns and how the revised paper answers them. The details are in the individual responses.

Strengths noted by the reviewers

  • The paper is well written, with clear motivation, contributions and methodology (gZMv), and its experiments and ablations are extensive (gZMv, PEmX).
  • The Optimization annotation pipeline uses recent foundation models to produce pseudo ground truth for contacts and forces from in-the-wild videos (2wGy).
  • ForceWall measures hand and foot forces, while most video-to-force work is evaluated on foot-floor forces only (PEmX, gZMv).
  • PACT is the best method on both climbing datasets and competitive on RICH (Huang et al., 2022), and its components are verified one by one (2wGy).

Main concerns and what we did

  1. How accurate are the training labels, and what does PACT add to them? (gZMv W1, Q2, Q5, Q6, PEmX W1, Q1). We integrate the evaluation of the Optimization, which produces our training labels (Tables 1, 2 and 4). On ForceWall the force MAE of the Optimization is 94.4 N, and PACT trained on these labels reaches 85.5 N (Table 2, ForceWall rows, column Avg.), showing generalization. The new Table 12 (Appendix E.3) runs the Optimization with three different input poses, from SAM 3D Body, from PACT and from the ground truth. It shows how much of the error of the Optimization comes from its kinematics and how much from its force estimation.
  2. Is the end-to-end model better than staged ones in a controlled comparison? (PEmX W2, Q2). The new Table 3 compares PACT with three staged models trained on the same labels. Two of them predict contacts and forces from the estimated motion, without access to the image, and the third is PhysPT (Zhang et al., 2024) retrained on our labels. PACT has the lowest force MAE on every limb and on average (87.5 N against 95.4 to 99.1 N for the staged models, Table 3, column Avg.), and the highest correlation with the sensors (0.729 against 0.609 to 0.667, column Corr).
  3. Is the evaluation independent of the Optimization? (2wGy W3, gZMv Q1, Q4). The ground truth for contacts and forces always comes from force sensors, force plates or manual annotation, and never from the Optimization. Thus, the two are completely independent. The new Figure 4 shows the pose, contacts and reconstructed 3D scene of the Optimization next to PACT's on in-the-wild videos.
  4. Is the main paper self-contained? (2wGy W4, gZMv Q7, Q8). The evaluation of the labels moved from the appendix to the main tables, and Section 4.3 now reports the runtime of the Optimization and of PACT.

Changes in the revised PDF (marked in blue)

  • Tables 1, 2 and 4 include the Optimization rows.
  • Table 3 (new) holds the staged models trained on our labels, described in Appendix C.6.
  • Figure 4 (new) compares the Optimization and PACT on in-the-wild videos.
  • Section 3.2 clarifies the kinematic objective, the role of the forces in Eq. 8 and how velocities and accelerations are computed.
  • Section 4.3 has a new runtime paragraph.
  • Appendix E.3 has the new Table 12 with its analysis, and Appendix C.5 describes the reference kinematics we fitted for OpenCap.

We hope the revision addresses the raised concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.