acceptodds
Under review as a conference paper at ICLR 2027

Videos to Forces: Physics Based Climbing Control From Online Videos

Abstract

Climbing motions are especially challenging to reconstruct from videos because of their dense, complex contacts. Existing video-based reconstructions recover kinematics but not the required dynamics; physics-based motion tracking with reinforcement learning (RL) offers an alternative, still relies on these imperfect kinematic references and requires the policy to discover viable contact forces through exploration. We reinterpret a fictitious force that prior methods use for partial assistance, instead using it to fully support the motion as an intermediate dynamic reference. Although residual assistance must ultimately be removed, this force reveals how much support must ultimately come from physical contacts, and we use it to densely guide the policy to learn to clime using only the physical contact forces. Our method reconstructs challenging climbing behaviors from video-derived motion references. Across diverse wall geometries and climbing techniques, it reduces penetration, improves motion quality, and makes policy learning more reliable and efficient than prior kinematic and physics-based approaches. To our knowledge, this is the first method to reconstruct complex human climbing motions from video using only physical contact forces at test time.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, character animation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_b5oZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a physics-based framework for reconstructing climbing motions from monocular videos. Its key idea is to first learn a motion policy fully supported by residual forces, use the learned forces to derive contact-force and joint-torque guidance, and gradually transfer the support to physical contacts. Experiments in Isaac Sim show improvements over DeepMimic and an unguided variant in motion tracking, contact loading, penetration, and training completion on several climbing motions.

Critical strengths

The submission addresses the difficult exploration landscape in physics-based climbing control with an intuitive key idea: leveraging residual forces as a source of dynamics supervision rather than conventional temporary scaffolding. The experimental setup is solid, offering meaningful ablation studies and demonstrating that the resulting policies operate with zero residual force at test time, thereby confirming full physical plausibility and contact reliance.

Critical weaknesses
  1. The evaluation is narrow, as the method relies on accurate wall geometry and trains a separate policy for each motion, without demonstrating generalization beyond individual motions.

  2. The baseline comparisons are insufficient to establish the method’s contribution. Comparisons with DeepMimic and a variant without dynamics guidance do not clarify whether the gains stem specifically from the learned intermediate dynamics reference or could be achieved by existing assistance-based curricula or contact-aware control methods.

  3. The comparison with JOSH focuses on kinematic reconstruction under different scene-geometry assumptions and does not address the lack of strong baselines for physics-based control.

  4. The submitted paper does not provide an accompanying demonstration video, making it difficult to assess the temporal quality, contact transitions, and stability of the reconstructed motions.

Questions for authors
  1. Can the authors compare their method against existing assistance-based curricula or contact-aware control methods under the same environment, reference motions, and zero-residual-force test conditions?

  2. Can the authors isolate the contribution of learning the fully assisted intermediate policy from that of the subsequent force-derived dynamics guidance? In particular, could simpler alternatives achieve comparable performance?

  3. Can the authors provide a demonstration video showing complete motions and baseline comparisons to better assess motion quality, contact transitions, and stability?

1 comment

I appreciate the authors' feedback. While I recognize the solid engineering implementation and impressive demo results, the current experiments remain somewhat limited in scope. Given its strengths, this work might be a better fit for a computer graphics (CG) conference.

Reviewer_eW4kOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning framework for recovering physically executable human motion from monocular rock climbing videos.

Critical strengths

I like the motivation for the paper very much, and I think that translating Residual Force into explicit dynamic guidance is substantively innovative. Especially when the world model explains physical conditions, this kind of work is very necessary.

Another point is that the scenes chosen by the author are very difficult, especially the experiments on complex rock climbing scenes, which are very challenging and convincing.

Critical weaknesses
  1. This article uses CoMotion to recover human motion from monocular video, and then combines it with the known three-dimensional mesh of the real rock wall to optimize the position, scale and contact posture of the human body. However, I think this article does not clearly explain how the real rock wall model is obtained, how the camera and rock wall coordinate systems are calibrated, and whether the initial correspondence between the human body and the hold is automatically calculated or provided manually. Since the title of the paper is "Videos to Forces", these details must be explained clearly. This information is crucial for rock climbing dynamics reconstruction. In particular, errors in the size of the rock wall, the location of holds and the global posture of the human body will directly change the contact position, moment arm and required joint torque. Without accurate scene geometry, even if RL can successfully complete the climb, it does not mean that it simulates the force exertion of real athletes. In addition, the paper lacks a systematic frame-by-frame comparison between the original RGB video and the final physical reconstruction results, nor does it fully quantify the deviation between the physically optimized human movements and the original video. I especially hope to see comparisons with real videos, which will allow me to fully judge the results of RL adjustment.
  2. This article guides RL to learn physical control through residual force, so that the simulated human body can finally complete rock climbing relying on the contact force of hands and feet. I think this idea is very valuable, but it needs to be distinguished between "complete the action in the simulation" and "accurately restore the force of the real athlete". These are two completely different concepts. The paper distributes the root residual force to multiple contact points through optimization, but this distribution is not necessarily unique, and there may be no feasible solution when there are few contact points or the friction conditions are limited. There is currently a lack of real measured contact force or joint torque as a basis for verification, so it is not yet certain whether the final mechanical results are consistent with the way the real human body generates force.
  3. We know that SMPL does not drive the fingers, and the visualization in the paper also shows that the five fingers are open and in contact with the holds. However, in rock climbing, there are often many finger points and precise finger skills are required. The author does not seem to explain this part.
Questions for authors
  1. The comparison between this article and JOSH seems unfair. JOSH needs to reconstruct unknown scenes at the same time, while this article directly uses the real rock wall Mesh. I hope the author can explain clearly what the setting of this experiment is.
  2. Referring to the first point of Weaknesses, I hope the author can explain: Is the real three-dimensional rock wall Mesh obtained through laser scanning, photogrammetry, manual modeling, or other methods? How to determine the initial transformation from the camera to the world coordinate system of the rock wall, the absolute scale and the corresponding relationship between the holds contact? Do these steps require manual calibration? Can the entire process be automated for a regular online video without pre-scanning the rock wall? If these cannot be completed automatically, would you consider changing the questions? Even change the idea of ​​​​the whole article and delete "Video", because the part about computing power is already very interesting.
  3. Can you provide frame-by-frame overlay visualizations of the original video, CoMotion reconstruction, contact-optimized reference motion, and final RL simulated motion, and report 2D keypoint reprojection errors or other motion fidelity metrics? I'd particularly like to see if the final simulated human body deviates significantly from the movements of a real athlete in order to satisfy physical contact constraints.
  4. The author trains RL Policy separately for each motion segment. Figure 5 uses a 48-hour training timeout limit, but does not report the actual training time, computing resource consumption, and complete convergence process of each action. I would like to see the actual training time, GPU model, total number of training steps and convergence of the two RL stages of each rock climbing movement in Figure 5. Also, can the parameters of existing Policy be reused, or generalized directly under new rock walls and contact conditions? For a "top" bouldering route, it may only take about 30 seconds. Does a 48-hour RL Policy require a 30-second climbing time? Is the computational cost of this move-by-movement approach too high?
  5. In addition, there are two rock climbing-related multi-modal data sets not mentioned in this article, namely CIMI4D (CVPR 2023) and AscendMotion (CVPR 2025). I hope the author can add them in the article.
Reviewer_mqDUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles reconstructing climbing motion from monocular video in a way that is physically supported by wall contacts rather than just kinematically plausible. The pipeline uses CoMotion to obtain 4D reconstruction, then refines the reconstruction against a ground-truth wall mesh with an optimization. Lastly, it trains motion imitation policies with PPO inside the physics simulator, Isaac Sim.

The core idea reverses the usual role of Residual Force Control. Rather than using the fictitious root wrench as a crutch during tracking, the authors first let it support the character entirely with collisions turned off, take the converged residual as the load that real contacts will eventually have to carry, and then anneal it to zero through a curriculum.

Critical strengths
  1. Climbing is a hard problem to pick due to dense and irregular contact geometries, and the paper deserves credit for taking it on. Therefore, it is a much tougher test for physics-based tracking than the locomotion benchmarks.

  2. The core idea is simple and I found it interesting. Treating the residual force as something to learn from rather than something to hide is a nice reframing.

  3. DeepMimic is a fair reference point for standard tracking, and Figure 7 is convincing that support gets handed over from the residual to actual contacts as training goes on.

Critical weaknesses
  1. The central claim is never checked. Everything downstream rests on the assumption that the converged residual is a faithful stand-in for the contact loading a real climber would need, but at no point does the paper compare any recovered force against ground truth. Before moving on to climbing, I would have liked to see the pipeline hold up on a motion where ground-truth dynamics exist, say walking or running with force-plate data (LAAS Parkour or ForcePose from [1] would both work), with an actual force error reported. The cartwheel in Fig. 8 goes some way toward this, but it only shows that the method does not fall apart on easy motions; it says nothing about whether the forces come out right. Without that kind of sanity check, I find the physical claims hard to credit.

  2. The paper could benefit from having more (obvious) baselines. The natural comparison is RFC itself with the residual kept at test time, since that would isolate what annealing it away actually buys. For the force-related claims, a video force-estimation method such as [1] would also be more relevant than DeepMimic, which was never meant to produce accurate contact forces in the first place.

  3. There is not enough detail to reproduce the method: no information on network architectures, PPO hyperparameters, the reward terms, the Isaac Sim contact parameters...

  4. The evaluation is small and loosely controlled. Results come from six climbs in Fig. 5, seven in Table 1 and four in Table 2, and the "harder / easier" split is never defined. Several of the 0.00 success rates are down to a 48-hour training timeout, which mixes up compute budget with method quality.

  5. The results are presented qualitatively for the most part. The qualitative evidence is hard to judge. For a motion paper, static figures do not tell me much about how natural the result looks or whether the contacts are believable. A supplementary video, or at least frame-by-frame sequences with the contact forces drawn in, would help a lot.

[1] Nathan Louis, Jason J. Corso, Tylan N. Templin, Travis D. Eliason, and Daniel P. Nicolella. Learning to Estimate External Forces of Human Motion in Video. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), 2022.

Questions for authors
  1. Can you validate the recovered forces on a dataset with measured ground reaction forces and report the force error?
  2. Can you provide videos, including the DeepMimic failure cases from Fig. 6?
  3. How are the per-frame contact sets and hold assignments determined?

Comments on the paper

2 comments
AuthorsOpenReview ↗

RFC baseline and force validation are now on our results website

We have updated our website (https://climb-anon.github.io/results/) with the two experiments promised in our first response:

  1. RFC baseline (mqDU\color{red}{mqDU}, b5oZ\color{ForestGreen}{b5oZ}), under "Comparisons". RFC uses the same curriculum that drives the residual force to zero as in our method, and we ran it on one segment from each of our 7 climbs.
  2. Force validation (mqDU\color{red}{mqDU}, eW4k\color{blue}{eW4k}), under "Force Validation". We compare against measured forces on 2 motions from the LAAS Parkour dataset, covering 1 to 4 contacts.

We’d like to make 3 points based on these experiments’ results.

Our policy gives a physically plausible solution of contact forces. We agree with eW4k\color{blue}{eW4k} that the split of total forces across contacts is not unique. Our method gives a single point in the solution space, and we do not claim it as the ground truth. How close the split is to the ground truth depends on how wide the solution space is. Following mqDU\color{red}{mqDU}’s suggestion, we track two MoCap sequences in the LAAS Parkour dataset, and compare our contact forces against its measurement. We apply our method with and without force guidance. The total force is constrained by the motion and is close to the measurement in both.

  1. Sequence 1 (Kong vault) has up to 4 contacts, so its solution space is wider. Without guidance, the hands push sideways against each other around frame 30. The measurement has no such forces, and our guidance removes them. This is because our minimum-norm objective (Eq. 4 in the paper) discourages forces that cancel each other. However, we do not recover the split between hands and feet around frame 17, where the athlete puts more weight on the feet. Eq. 4 cannot capture the athlete's strategy or the strength difference across limbs. We will state this as a limitation.
  2. Sequence 2 (safety vault) has 1 or 2 contacts throughout, so its solution space is narrow. Here our per-end effector forces are close to the measurement with and without guidance.

Our method tracks motions that prior methods cannot track or track poorly. Prior methods follow the kinematic reference, including its errors, and must discover the contact forces by exploration. Our guidance instead tells the policy how to load each contact. In the RFC comparison, within the training budget, RFC cannot track Men 1, Men 2, Women 4, and tracks Men 3 poorly. The safety vault gives a plausible explanation in a more controlled setup. Its MoCap reference has a small left-foot penetration into the bar at frame 17. Without guidance, our policy tries to follow the reference and results in a twisted leg pose in frames 16-25; with guidance our policy gives natural leg poses. Our explanation is that in early training when the foot taps the bar, the force guidance leads it to avoid the bar instead of following the reference into it. Such reference errors are more common in video reconstruction data than in MoCap, which could explain why our guided policy has consistent performance on our climbs.

Where prior methods can track, our force split is arguably more reasonable. An example is at the 2-3 seconds of Women 2’s RFC comparison. There RFC gives large forces in almost opposite directions on both hands, while our method gives much smaller forces by asking both force directions to be upward. Another example is the opening frames of Men 1 where the climber hangs on a hold with both arms. Our method consistently gives upward forces on both hands, while RFC can give one upward and one downward force. Both are consistent with our Kong vault result, where our method eliminates the cancelling x-direction forces on the hands.

Misc. In the force validation, our total force has two differences from the ground truth. In the Kong vault, when only the hands are on the bar, our forces are lower than measured; we believe this comes from small tracking errors in the pose. In the safety vault, the spike in the first few frames is likely an initialization artifact, because the clip starts mid-motion.

AuthorsOpenReview ↗

Summary of paper revisions

We have uploaded a revised paper. The changes are:

  1. Introduction (Sec. 1) (mqDU\color{red}{mqDU}). Some small wording edits so that the text does not suggest we recover the athletes' forces.
  2. Related Work (Sec. 2) (eW4k\color{blue}{eW4k}). We now cite CIMI4D and AscendMotion.
  3. Method (Sec. 3) (mqDU\color{red}{mqDU}, eW4k\color{blue}{eW4k}). We state that the force split is not unique, and that our minimum-norm objective selects one physically valid solution instead of an estimate of the climber's actual forces.
  4. Results (Sec. 4).
    • We add a link to our results website.
    • We report the GPU model and training budget, with per-motion training times in Appendix A.3 (eW4k\color{blue}{eW4k}).
    • The "easier" and "harder" descriptions of motions are removed (mqDU\color{red}{mqDU}).
    • The hand model and the friction coefficient are stated in the setup (mqDU\color{red}{mqDU}, eW4k\color{blue}{eW4k}).
  5. Limitations (Sec. 6) (mqDU\color{red}{mqDU}, eW4k\color{blue}{eW4k}). We added two new paragraphs discussing non-unique contact forces and simplified hand contact.
  6. Appendix A.1 (mqDU\color{red}{mqDU}, eW4k\color{blue}{eW4k}). More details on our data processing steps.

We thank the reviewers again for their feedback, which has made our paper clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.