- Rating
- 2
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Latent Jacobian Steer (LJS), an offline method for generating contact-rich behaviors with a frozen humanoid controller. LJS maps joint-action proposals into latent commands through a damped inverse of the policy Jacobian and evaluates them using simulator rollouts within CEM search. Tasks are specified as sequences of human-defined physical effects. On three MuJoCo tasks using BFM-Zero, LJS achieves higher mean stage completion than the evaluated baselines without task-specific demonstrations or policy fine-tuning.
Critical strengths
- Valuable problem and a simple, well-motivated method. LJS addresses how to direct pretrained motor capabilities toward new contact tasks. Coupling joint-action proposals with latent-command execution provides a clear interface for obtaining precise interactions while retaining the frozen controller.
- Relevant empirical comparisons. Direct latent search and joint-residual search test the central design choice, while optimizer and random-subspace ablations provide further support. LJS achieves higher mean stage completion across all three tasks. The shutter ablation also suggests that dimensionality reduction alone is insufficient to recover its performance.
Critical weaknesses
- Limited evidence of reliability. Main results use only three seeds per task, with LJS completing 10, 10, and 1 stages on hurdle traversal. The detailed ablations use a single seed on the shutter task. This leaves substantial uncertainty about task success rates and whether the ablation conclusions hold across runs and tasks.
- Limited validation of practical reuse. Physical-effect definitions are refined through rollout inspection, but the paper does not systematically test whether the same definitions remain effective under changes in the initial robot–object configuration. It is therefore unclear whether such changes require further manual adjustment. Hardware execution and the proposed downstream learning uses are also untested. Although acknowledged as future work, these gaps leave the method’s robustness and practical value insufficiently demonstrated.
Questions for authors
- What caused the hurdle run to stop after the first effect? Additional seeds for the main comparisons and key ablations would help assess the consistency of the reported gains.
- Do the reported seeds change the initial robot–object configuration or only the search randomness? Can LJS still succeed after modest changes in the robot’s initial position or orientation, while keeping the effect definitions fixed and rerunning search?
- Some demonstrated motions appear to involve large movements. Has their feasibility on physical hardware been assessed, and would they require modification before real-world execution?
7 comments
Response to Reviewer sHhd (2/4)
We also demonstrate the generality of the method by integrating CEM-LJS with another general pretrained controller on the Sharpa-Franka setup used in Play2Perfect [a]. The steered policy in that case is the Play2Perfect pretrained 6-dimensional goal-conditioned policy - observe that the goal (g) can now be thought of loosely as a search variable to sequence/compose and steer the policy over the whole episode. Our baseline in this case is the direct-goal search with CEM on the Play2Perfect policy. We report results for 10 seeds across four out-of-distribution assembly tasks. The Play2Perfect authors had to fine-tune the policy task-specifically to handle these exact four OOD tasks. However, we do not fine-tune or re-train and only use the pretrained controller for searching and eliciting the assembly behaviors. As demonstrated, CEM-LJS performs the best in this scenario as well.
Table: Assembly tasks with the frozen Play2Perfect policy, ten seeds (paper Table 2, Appendix K). The policy is used as pretrained, without any fine-tuning. The first two columns give the number of fully completed tasks, and the last two give the search time of the completed runs, mean with the fastest in parentheses.
| Task | CEM-LJS | CEM direct-goal | Search time, CEM-LJS | Search time, CEM direct-goal |
|---|---|---|---|---|
| Beam assembly, first part | 10/10 | 10/10 | 62 s (29 s) | 46 s (29 s) |
| Beam assembly, second part | 7/10 | 4/10 | 72 s (43 s) | 65 s (48 s) |
| Screwing | 9/10 | 3/10 | 93 s (52 s) | 132 s (88 s) |
| Insertion | 5/10 | 4/10 | 83 s (69 s) | 64 s (22 s) |
| All | 31/40 | 21/40 |
Finally, to demonstrate the ease of use of our method on any steerable latent- or goal-conditioned pretrained controller, we also show its transfer to FB-MEBE [c], ASE [d], and SPiRL [e] policies, and we also run BFM-Zero with LJS in a second simulator, Isaac Sim (picking, carrying, and placing a box). We don’t make any comparisons for this in the interest of time, but we expect most baseline methods will also work here, since these OOD tasks are very simple and are just showcased as a demo. The rollouts from these can be viewed on our publicly available, anonymous website, and more details are provided in our updated Appendix L.
Q1: Concerns about the hurdle scores
Our explanation, based on our analysis of the failed hurdle run, is that the CEM search method sometimes gets stuck in basins or bad local optima. For example, in the run mentioned in the paper, the humanoid placed its palms on the top of the hurdle but stalled while bringing its right leg onto the top, with its knee pressed against the near face of the hurdle below its edge. We hypothesize that when hyper-precise movements drive the CEM-LJS searches into a bad basin or a local optimum, it performs worse than the baselines on rare occasions.
How can we fix this? The simple robustness fix is to do a bounded number of retries from slightly earlier search points when stuck at some stage of the search. We don’t advertise this as the main method, as we only wanted to compare the efficacy of the latent Jacobian pullback. However, for completeness and practical usage, we run this modified version of CEM for LJS, direct-z, and residual-actions on BFM. We show that CEM-LJS achieves near-100% success with this retry-modified CEM search, at the cost of a higher average search wall-clock time (which is still very reasonable compared to other methods).
Table: Search with retries and randomized physics, ten new seeds per task (paper Appendix I). If an effect fails, the search tries it again, and if it fails again, it goes back and searches the previous effect again. Every candidate is scored under four randomly drawn physics settings (friction, link masses, PD gains and control delay), and each run is limited to 15 minutes. Entries give the number of fully completed tasks, and the last column gives the mean search time of CEM-LJS.
| Task | CEM-LJS | CEM direct-z | CEM residual-BFM | Mixed-Gaussian LJS | Mean search time, CEM-LJS |
|---|---|---|---|---|---|
| Hurdle (10 effects) | 10/10 | 6/10 | 7/10 | 2/10 | 479 s |
| Shutter (5 effects) | 10/10 | 6/10 | 0/10 | 3/10 | 328 s |
| Door + cube (6 effects) | 10/10 | 10/10 | 2/10 | 10/10 | 145 s |
| All | 30/30 | 22/30 | 9/30 | 15/30 |
Response to Reviewer sHhd (3/4)
W2(a), Q2: Randomized starting configurations with fixed physical effect scores
The random seeds reported so far indeed had identical starts. To alleviate concerns about robustness given the same fixed physical effect scores, we use both position- and orientation-randomized starts across 10 seeds on the shutter task and report our method's results, while keeping the physical effect definitions fixed. CEM-LJS remains the best on this as well.
Table: Shutter from ten moved and turned starting positions (paper Appendix I). Each start is moved by 0.05–0.19 m and turned by −11.2° to +11.8° from the default start, with the effect definitions unchanged. Search times are for the completed runs, mean with the fastest in parentheses.
| Method | Shutters completed | Search time |
|---|---|---|
| CEM-LJS | 8/10 | 84 s (57 s) |
| CEM direct-z | 1/10 | 158 s (158 s) |
| CEM residual-BFM | 0/10 | – |
| Mixed-Gaussian LJS | 2/10 | 95 s (93 s) |
We also run a new sanity-check task of carrying a 3 kg box through 10 points in 3D space, where a single minimal physical-effect definition (object goals and a constraint that the box never touches the floor) is reused unchanged across three generated paths. CEM-LJS completes 12/15 runs, while the best baseline completes 3/15. (Please refer to Table 3 and Appendix J of the updated paper or the website linked for more details on this if needed).
W2(b), Q3: Hardware concerns
Since we had enough time to address this issue as well, we did two things: (1) observe the motion quality metrics and analyze them across the methods for the existing set of search rollouts, (2) as per the final suggestion you provided, we also did a separately modified set of searches on the hurdle task (with a quick glance at the videos, the most aggressive motions are seen here). For this, instead of letting a downstream RL or learning method correct and regularize these motions, we solve the problem at its roots by adding a human-likeness regularization term to the search’s physical effect scores. We note that this is only one of many ways to adapt our method to improve its sim2real robustness.
(1) We show, with the following metrics, how CEM-LJS in general is a much more physically agreeable search method than searching in direct joint-action space.
Table: Motion quality of CEM-LJS and CEM residual-BFM (paper Appendix H). We compare the 65 effects that both methods completed on the same task and seed, using medians and a two-sided Wilcoxon signed-rank test. The limits are those of the Unitree G1's joints and motors. Human-likeness is the score of BFM-Zero's own motion discriminator, which neither method optimizes.
| Measure (per effect) | CEM-LJS | CEM residual-BFM | Effects where CEM-LJS is better | p-value |
|---|---|---|---|---|
| Mechanical power (W/kg) | 1.59 | 2.61 | 56/65 | 6e-07 |
| Joint jerk, RMS (rad/s³) | 782 | 1660 | 54/65 | 1e-07 |
| Peak commanded torque / motor limit | 1.28 | 2.53 | 54/65 | 4e-08 |
| Time at the motor torque limit | 0.06 | 0.20 | 43/65 | 9e-03 |
| Joint-target rate, RMS (rad/s) | 2.23 | 4.03 | 55/65 | 1e-07 |
| Peak base angular speed (rad/s) | 2.93 | 4.05 | 58/65 | 3e-10 |
| Peak joint speed / G1 limit | 0.33 | 0.50 | 54/65 | 5e-08 |
| Excursion beyond G1 joint range (rad) | 0.005 | 0.025 | 55/65 | 1e-09 |
| Foot slip under load (m/s) | 0.020 | 0.112 | 55/65 | 2e-08 |
| Time with self-contact | 0.24 | 0.42 | 44/65 | 2e-04 |
| Deepest interpenetration (m) | 0.008 | 0.011 | 41/65 | 2e-02 |
| Peak contact force (body weights) | 3.03 | 3.47 | 35/65 | 0.3 |
| Human-likeness (higher is more human) | −0.89 | −2.40 | 57/65 | 5e-10 |
We also note that no run of any method in the ten-seed evaluation exceeds the G1's joint velocity limits. The commanded torque, however, does peak above the motor limit for every method (a median peak of 1.3–2.1× the limit per task). Because this also happens for CEM direct-z, which only changes BFM-Zero's latent command, the peaks come from BFM-Zero's own PD tracking during contact rather than from LJS. Whenever the commanded torque exceeds a motor's limit, the motor simply applies its maximum torque instead, as a real motor would (paper Appendix H).
(2) We use the discriminator provided by BFM-Zero [b] itself as our human-likeness regularizer and run 10 seeds of retry-modified CEM-LJS with and without the regularizer. This shows that, with only small regularization changes to the physical effect scores while keeping the rest fixed, we can obtain more physically agreeable motions.
Response to Reviewer sHhd (1/4)
We thank the reviewer for appreciating the value of the problem we sought to solve, the simplicity of our novel method, and the relevance of our empirical comparisons. We address the critical weaknesses mentioned and questions asked by (1) increasing the breadth (4 more tasks in another embodiment) and depth (10 new seeds) of empirical evidence, (2) showing that our method, with the same fixed physical effect definitions, generalizes to randomized physics and randomized initial configurations, (3) identifying why some hurdle searches fail, and suggesting and validating a simple, general search trick that increases the success of CEM-LJS to near-100%, and (4) providing physical feasibility metrics in simulation and showing that CEM-LJS, even without regularization, is already largely physically agreeable as opposed to searching in just direct action space. Our conclusions that, due to action-latent coupled exploration, CEM-LJS outperforms and is more robust than all other baselines still hold. We hope that these strengthen our claim and address the reviewer’s concerns satisfactorily.
To make it easy for reviewers to access all videos and results in one place, we also link to the anonymous LJS website: https://latent-jacobian-steer.github.io/. Following the ICLR guidelines, the website is fully anonymous and does not track its visitors.
W1: More empirical evidence, along with wall-clock search-time metrics
We now report 10 seeds per task, along with the search ablations on all three humanoid tasks. CEM-LJS consistently outperforms all baselines in mean effect scores across all humanoid tasks. For this, we used completely unseen seeds and kept the physical effects fixed (note that we added a final, stable standing effect to the shutter task, bringing the total to 5 effects). The other two tasks have 10 effects (hurdle) and 6 effects (door-cube). To avoid any method-specific tuning, we also kept a fixed pair of simulation and execution horizons (40 and 20 control steps, respectively). We kept the search budgets and horizons fixed across methods and seeds, and ensured that the number of search decisions allowed for all effects is at most 30. We also note that most successful CEM-LJS searches finish within 1–3 minutes, which makes it practical as a quick search method for behaviors.
Table: Humanoid tasks with BFM-Zero, ten new seeds (paper Table 1). Each entry gives the mean number of physical effects completed ± standard deviation, with the number of fully completed tasks in parentheses.
Table: Wall-clock search time of the runs that completed the task (paper Appendix F). Mean, with the fastest run in parentheses, measured on one RTX 4090 with one search running at a time. A dash means the method never completed that task.
Table: Search ablations on all three humanoid tasks, ten seeds each (paper Appendix G). Each entry gives the number of fully completed tasks, followed by the mean number of effects completed. The first six rows propose joint-action changes through the Jacobian (LJS), and the last five search in the latent space directly.