acceptodds
Under review as a conference paper at ICLR 2027

Latent Jacobian Steer: Eliciting Complex Contact-Rich Behaviors from Pretrained Humanoid Controllers

Abstract

Given a pretrained general latent-conditioned whole-body humanoid controller, how can we elicit complex and contact-rich out-of-distribution behaviors? Joint-action exploration enables direct task-specific adjustments but can disrupt coordination learned by the pretrained controller, whereas latent-command exploration retains it but is difficult to steer toward desired physical effects. To combine the complementary strengths of these exploration spaces, we propose Latent Jacobian Steer (LJS), which couples sampling in direct joint-action space with the semantics of the learned frozen controller's input latent space. For a task specified through desired physical effects, we sample task-relevant joint-action perturbations and map them through a damped local inverse of the policy Jacobian to obtain candidate latent commands. These commands are executed through the frozen pretrained policy, while exploration is directed toward the physical outcomes required by the new task. Candidate outcomes are evaluated in simulation and composed into latent-command sequences that accomplish the task. Ablations show that neither action-space nor latent-space exploration alone provides the same capability as their coupling. Across diverse out-of-distribution tasks, test-time searching using this method produces contact-rich behaviors, including hurdling, manipulating articulated objects, pivoting large objects against the environment, and opening doors through non-prehensile contact.

Primary area:
applications to robotics, autonomy, planning
Keywords:
humanoid control, whole-body locomanipulation, exploration, behavioral foundation models, robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sHhdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Latent Jacobian Steer (LJS), an offline method for generating contact-rich behaviors with a frozen humanoid controller. LJS maps joint-action proposals into latent commands through a damped inverse of the policy Jacobian and evaluates them using simulator rollouts within CEM search. Tasks are specified as sequences of human-defined physical effects. On three MuJoCo tasks using BFM-Zero, LJS achieves higher mean stage completion than the evaluated baselines without task-specific demonstrations or policy fine-tuning.

Critical strengths
  1. Valuable problem and a simple, well-motivated method. LJS addresses how to direct pretrained motor capabilities toward new contact tasks. Coupling joint-action proposals with latent-command execution provides a clear interface for obtaining precise interactions while retaining the frozen controller.
  2. Relevant empirical comparisons. Direct latent search and joint-residual search test the central design choice, while optimizer and random-subspace ablations provide further support. LJS achieves higher mean stage completion across all three tasks. The shutter ablation also suggests that dimensionality reduction alone is insufficient to recover its performance.
Critical weaknesses
  1. Limited evidence of reliability. Main results use only three seeds per task, with LJS completing 10, 10, and 1 stages on hurdle traversal. The detailed ablations use a single seed on the shutter task. This leaves substantial uncertainty about task success rates and whether the ablation conclusions hold across runs and tasks.
  2. Limited validation of practical reuse. Physical-effect definitions are refined through rollout inspection, but the paper does not systematically test whether the same definitions remain effective under changes in the initial robot–object configuration. It is therefore unclear whether such changes require further manual adjustment. Hardware execution and the proposed downstream learning uses are also untested. Although acknowledged as future work, these gaps leave the method’s robustness and practical value insufficiently demonstrated.
Questions for authors
  1. What caused the hurdle run to stop after the first effect? Additional seeds for the main comparisons and key ablations would help assess the consistency of the reported gains.
  2. Do the reported seeds change the initial robot–object configuration or only the search randomness? Can LJS still succeed after modest changes in the robot’s initial position or orientation, while keeping the effect definitions fixed and rerunning search?
  3. Some demonstrated motions appear to involve large movements. Has their feasibility on physical hardware been assessed, and would they require modification before real-world execution?
7 comments
AuthorsOpenReview ↗

Response to Reviewer sHhd (1/4)

We thank the reviewer for appreciating the value of the problem we sought to solve, the simplicity of our novel method, and the relevance of our empirical comparisons. We address the critical weaknesses mentioned and questions asked by (1) increasing the breadth (4 more tasks in another embodiment) and depth (10 new seeds) of empirical evidence, (2) showing that our method, with the same fixed physical effect definitions, generalizes to randomized physics and randomized initial configurations, (3) identifying why some hurdle searches fail, and suggesting and validating a simple, general search trick that increases the success of CEM-LJS to near-100%, and (4) providing physical feasibility metrics in simulation and showing that CEM-LJS, even without regularization, is already largely physically agreeable as opposed to searching in just direct action space. Our conclusions that, due to action-latent coupled exploration, CEM-LJS outperforms and is more robust than all other baselines still hold. We hope that these strengthen our claim and address the reviewer’s concerns satisfactorily.

To make it easy for reviewers to access all videos and results in one place, we also link to the anonymous LJS website: https://latent-jacobian-steer.github.io/. Following the ICLR guidelines, the website is fully anonymous and does not track its visitors.

W1: More empirical evidence, along with wall-clock search-time metrics

We now report 10 seeds per task, along with the search ablations on all three humanoid tasks. CEM-LJS consistently outperforms all baselines in mean effect scores across all humanoid tasks. For this, we used completely unseen seeds and kept the physical effects fixed (note that we added a final, stable standing effect to the shutter task, bringing the total to 5 effects). The other two tasks have 10 effects (hurdle) and 6 effects (door-cube). To avoid any method-specific tuning, we also kept a fixed pair of simulation and execution horizons (40 and 20 control steps, respectively). We kept the search budgets and horizons fixed across methods and seeds, and ensured that the number of search decisions allowed for all effects is at most 30. We also note that most successful CEM-LJS searches finish within 1–3 minutes, which makes it practical as a quick search method for behaviors.

Table: Humanoid tasks with BFM-Zero, ten new seeds (paper Table 1). Each entry gives the mean number of physical effects completed ± standard deviation, with the number of fully completed tasks in parentheses.

MethodHurdle (10 effects)Shutter (5 effects)Door + cube (6 effects)
CEM-LJS6.20 ± 3.43 (3/10)3.90 ± 2.08 (7/10)6.00 ± 0.00 (10/10)
CEM direct-z4.60 ± 2.99 (1/10)0.10 ± 0.32 (0/10)4.90 ± 1.45 (6/10)
CEM residual-BFM5.30 ± 3.92 (3/10)1.50 ± 1.08 (0/10)1.70 ± 1.25 (0/10)
Mixed-Gaussian LJS3.60 ± 3.53 (1/10)0.10 ± 0.32 (0/10)4.10 ± 2.47 (5/10)

Table: Wall-clock search time of the runs that completed the task (paper Appendix F). Mean, with the fastest run in parentheses, measured on one RTX 4090 with one search running at a time. A dash means the method never completed that task.

MethodHurdleShutterDoor + cube
CEM-LJS137 s (104 s)77 s (54 s)61 s (55 s)
CEM direct-z152 s (152 s)–72 s (58 s)
CEM residual-BFM89 s (80 s)––
Mixed-Gaussian LJS189 s (189 s)–82 s (62 s)

Table: Search ablations on all three humanoid tasks, ten seeds each (paper Appendix G). Each entry gives the number of fully completed tasks, followed by the mean number of effects completed. The first six rows propose joint-action changes through the Jacobian (LJS), and the last five search in the latent space directly.

SearchShutter (5 effects)Hurdle (10 effects)Door + cube (6 effects)
CEM-LJS7/10, 3.93/10, 6.210/10, 6.0
MPPI [f] LJS4/10, 2.02/10, 6.39/10, 5.4
LATTICE-style [g] LJS3/10, 1.51/10, 3.67/10, 4.5
Zero-mean refinement LJS2/10, 1.01/10, 4.42/10, 2.1
Targeted-mean refinement LJS1/10, 0.50/10, 3.64/10, 4.2
Gaussian-only LJS0/10, 0.11/10, 5.52/10, 3.0
CEM direct-z0/10, 0.11/10, 4.66/10, 4.9
MPPI [f] direct-z3/10, 1.50/10, 2.36/10, 4.8
Gaussian direct-z0/10, 0.00/10, 2.74/10, 4.4
Targeted refinement direct-z0/10, 0.00/10, 1.63/10, 3.6
Random 29-D latent subspace0/10, 0.00/10, 0.80/10, 1.9
AuthorsOpenReview ↗

Response to Reviewer sHhd (2/4)

We also demonstrate the generality of the method by integrating CEM-LJS with another general pretrained controller on the Sharpa-Franka setup used in Play2Perfect [a]. The steered policy in that case is the Play2Perfect pretrained 6-dimensional goal-conditioned policy - observe that the goal (g) can now be thought of loosely as a search variable to sequence/compose and steer the policy over the whole episode. Our baseline in this case is the direct-goal search with CEM on the Play2Perfect policy. We report results for 10 seeds across four out-of-distribution assembly tasks. The Play2Perfect authors had to fine-tune the policy task-specifically to handle these exact four OOD tasks. However, we do not fine-tune or re-train and only use the pretrained controller for searching and eliciting the assembly behaviors. As demonstrated, CEM-LJS performs the best in this scenario as well.

Table: Assembly tasks with the frozen Play2Perfect policy, ten seeds (paper Table 2, Appendix K). The policy is used as pretrained, without any fine-tuning. The first two columns give the number of fully completed tasks, and the last two give the search time of the completed runs, mean with the fastest in parentheses.

TaskCEM-LJSCEM direct-goalSearch time, CEM-LJSSearch time, CEM direct-goal
Beam assembly, first part10/1010/1062 s (29 s)46 s (29 s)
Beam assembly, second part7/104/1072 s (43 s)65 s (48 s)
Screwing9/103/1093 s (52 s)132 s (88 s)
Insertion5/104/1083 s (69 s)64 s (22 s)
All31/4021/40

Finally, to demonstrate the ease of use of our method on any steerable latent- or goal-conditioned pretrained controller, we also show its transfer to FB-MEBE [c], ASE [d], and SPiRL [e] policies, and we also run BFM-Zero with LJS in a second simulator, Isaac Sim (picking, carrying, and placing a box). We don’t make any comparisons for this in the interest of time, but we expect most baseline methods will also work here, since these OOD tasks are very simple and are just showcased as a demo. The rollouts from these can be viewed on our publicly available, anonymous website, and more details are provided in our updated Appendix L.

Q1: Concerns about the hurdle scores

Our explanation, based on our analysis of the failed hurdle run, is that the CEM search method sometimes gets stuck in basins or bad local optima. For example, in the run mentioned in the paper, the humanoid placed its palms on the top of the hurdle but stalled while bringing its right leg onto the top, with its knee pressed against the near face of the hurdle below its edge. We hypothesize that when hyper-precise movements drive the CEM-LJS searches into a bad basin or a local optimum, it performs worse than the baselines on rare occasions.

How can we fix this? The simple robustness fix is to do a bounded number of retries from slightly earlier search points when stuck at some stage of the search. We don’t advertise this as the main method, as we only wanted to compare the efficacy of the latent Jacobian pullback. However, for completeness and practical usage, we run this modified version of CEM for LJS, direct-z, and residual-actions on BFM. We show that CEM-LJS achieves near-100% success with this retry-modified CEM search, at the cost of a higher average search wall-clock time (which is still very reasonable compared to other methods).

Table: Search with retries and randomized physics, ten new seeds per task (paper Appendix I). If an effect fails, the search tries it again, and if it fails again, it goes back and searches the previous effect again. Every candidate is scored under four randomly drawn physics settings (friction, link masses, PD gains and control delay), and each run is limited to 15 minutes. Entries give the number of fully completed tasks, and the last column gives the mean search time of CEM-LJS.

TaskCEM-LJSCEM direct-zCEM residual-BFMMixed-Gaussian LJSMean search time, CEM-LJS
Hurdle (10 effects)10/106/107/102/10479 s
Shutter (5 effects)10/106/100/103/10328 s
Door + cube (6 effects)10/1010/102/1010/10145 s
All30/3022/309/3015/30
AuthorsOpenReview ↗

Response to Reviewer sHhd (3/4)

W2(a), Q2: Randomized starting configurations with fixed physical effect scores

The random seeds reported so far indeed had identical starts. To alleviate concerns about robustness given the same fixed physical effect scores, we use both position- and orientation-randomized starts across 10 seeds on the shutter task and report our method's results, while keeping the physical effect definitions fixed. CEM-LJS remains the best on this as well.

Table: Shutter from ten moved and turned starting positions (paper Appendix I). Each start is moved by 0.05–0.19 m and turned by −11.2° to +11.8° from the default start, with the effect definitions unchanged. Search times are for the completed runs, mean with the fastest in parentheses.

MethodShutters completedSearch time
CEM-LJS8/1084 s (57 s)
CEM direct-z1/10158 s (158 s)
CEM residual-BFM0/10–
Mixed-Gaussian LJS2/1095 s (93 s)

We also run a new sanity-check task of carrying a 3 kg box through 10 points in 3D space, where a single minimal physical-effect definition (object goals and a constraint that the box never touches the floor) is reused unchanged across three generated paths. CEM-LJS completes 12/15 runs, while the best baseline completes 3/15. (Please refer to Table 3 and Appendix J of the updated paper or the website linked for more details on this if needed).

W2(b), Q3: Hardware concerns

Since we had enough time to address this issue as well, we did two things: (1) observe the motion quality metrics and analyze them across the methods for the existing set of search rollouts, (2) as per the final suggestion you provided, we also did a separately modified set of searches on the hurdle task (with a quick glance at the videos, the most aggressive motions are seen here). For this, instead of letting a downstream RL or learning method correct and regularize these motions, we solve the problem at its roots by adding a human-likeness regularization term to the search’s physical effect scores. We note that this is only one of many ways to adapt our method to improve its sim2real robustness.

(1) We show, with the following metrics, how CEM-LJS in general is a much more physically agreeable search method than searching in direct joint-action space.

Table: Motion quality of CEM-LJS and CEM residual-BFM (paper Appendix H). We compare the 65 effects that both methods completed on the same task and seed, using medians and a two-sided Wilcoxon signed-rank test. The limits are those of the Unitree G1's joints and motors. Human-likeness is the score of BFM-Zero's own motion discriminator, which neither method optimizes.

Measure (per effect)CEM-LJSCEM residual-BFMEffects where CEM-LJS is betterp-value
Mechanical power (W/kg)1.592.6156/656e-07
Joint jerk, RMS (rad/s³)782166054/651e-07
Peak commanded torque / motor limit1.282.5354/654e-08
Time at the motor torque limit0.060.2043/659e-03
Joint-target rate, RMS (rad/s)2.234.0355/651e-07
Peak base angular speed (rad/s)2.934.0558/653e-10
Peak joint speed / G1 limit0.330.5054/655e-08
Excursion beyond G1 joint range (rad)0.0050.02555/651e-09
Foot slip under load (m/s)0.0200.11255/652e-08
Time with self-contact0.240.4244/652e-04
Deepest interpenetration (m)0.0080.01141/652e-02
Peak contact force (body weights)3.033.4735/650.3
Human-likeness (higher is more human)−0.89−2.4057/655e-10

We also note that no run of any method in the ten-seed evaluation exceeds the G1's joint velocity limits. The commanded torque, however, does peak above the motor limit for every method (a median peak of 1.3–2.1× the limit per task). Because this also happens for CEM direct-z, which only changes BFM-Zero's latent command, the peaks come from BFM-Zero's own PD tracking during contact rather than from LJS. Whenever the commanded torque exceeds a motor's limit, the motor simply applies its maximum torque instead, as a real motor would (paper Appendix H).

(2) We use the discriminator provided by BFM-Zero [b] itself as our human-likeness regularizer and run 10 seeds of retry-modified CEM-LJS with and without the regularizer. This shows that, with only small regularization changes to the physical effect scores while keeping the rest fixed, we can obtain more physically agreeable motions.

Reviewer_oBf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Latent Jacobian Steer which produces contact-rich behaviors from a frozen humanoid controller without task-specific policy training or reference motions. The paper uses CEM proposes joint-action perturbations over the orignal proposal, which a damped inverse of the policy Jacobian maps into latent commands. Simulator rollouts evaluate these commands against human defined stages. LJS is used to produce behavior of hurdle traversal, shutter traversal, and door opening and carrying a cube, which are not trained by the controller BFM-zero, and this outperforming direct latent and residual-action search.

Critical strengths

The paper introduce a technically sound interface for reusing pretrained controller for novel use. The Jacobian mapping connects interpretable action changes to execution concretely. Simulation evaluates the resulting contact dynamics, avoiding any requirement for differentiable physics.

The paper compares direct latent search, residual-action search, and LJS under several tasks. LJS consistently performs better.

Critical weaknesses

The qualitative results leave concerns about the quality of the generated motions, while evaluation primarily measures completion of predefined physical effects. There is no systematic assessment of smoothness, stability, actuator demands, or robustness. Consequently, successful simulated task completion does not establish that these behaviors would provide useful, physically transferable demonstrations that can be deployed in the real world. From the demo video, the quality remains low and some motion are probably too aggressive.

Although the method targets offline generation in sim, this uncertainty limits its practical significance.

The benchmark is too narrow to substantiate the breadth of the claims. Evaluation covers only three manually staged tasks with three seeds each, and the detailed ablations use one shutter-task seed. These examples demonstrate selected interaction capabilities, but provide limited evidence across different geometries, contact conditions, and initial configurations. The concern is whether LJS reliably discovers diverse contact-rich behaviors beyond these carefully developed scenarios. Given that I feel "Complex Contact-Rich Behaviors" is a bit overclaimed.

Questions for authors

The method requires dedicated stage decompositions, cost functions, and transition conditions, refined through rollout inspection. While manageable for the demonstrated tasks, it is unclear how this engineering effort scales to longer behaviors involving more objects and complex contacts. Additional intermediate stages and revisions to earlier objectives may increase human labor, limiting the scalability of the pipeline.

Adding cost terms for motion smoothness (e.g., jerk minimization), energy efficiency, and similarity to natural human motion could improve motion quality and potentially facilitate sim-to-real transfer, although the latter would require further validation.

8 comments
AuthorsOpenReview ↗

Response to Reviewer oBf8 (1/5)

We thank the reviewer for appreciating the soundness of the interface we introduced and for taking the time to provide such constructive feedback. We summarize how we address concerns about the lack of systematic assessment of physical feasibility metrics, engineering effort, scalability, and the breadth of experiments:

In terms of effort and scalability, we present our arguments about why this approach is highly effective for generating novel OOD behavior and collecting data.

We also show, with new physical feasibility-related metrics, that CEM-LJS has greater agreement with hardware limits than residual action search. We provide human-likeness scores using the exact discriminator used in the BFM-Zero [a] work itself and show that, even without regularization, CEM-LJS generally yields more human-like and physically agreeable behaviors than a search method that only searches in the action space. We stress this because we believe this would be one of the first search methods to respect feasible dynamics rather than just optimizing for the task objective, despite admittedly yielding less smooth trajectories. We also took your suggestion to add such a human-likeness regularization to the objective itself and conduct a controlled experiment on the hurdle task to show how to further improve sim2real feasibility, and positively affect smoothness.

Finally, to broaden our empirical evidence, we increase its breadth (4 more tasks in another embodiment) and depth (10 new seeds).

To make it easy for reviewers to access all videos and results in one place, we link to the anonymous LJS website: https://latent-jacobian-steer.github.io/. Following the ICLR guidelines, the website is fully anonymous and does not track its visitors.

Q1: Engineering effort and the practicality of our search method

We would like to stress that CEM-LJS is one of the first methods to successfully steer pretrained latent-conditioned policies to exhibit out-of-distribution behaviors without requiring teleoperation, mocap, or similar methods. The prior methods also generally exhibit retargeting-based errors or other sources of noise that could be corrected using a learned, regularized model in simulation before transfer to real life. However, we still note the ease of engineering and the practical benefits of using our method with the following points:

  1. We conducted a new set of evaluations on 10 previously unseen seeds for several tasks, including tasks over another controller on another embodiment. Due to the efficiency and optimized engineering of our plug-and-play-like pipeline, as well as fast simulation parallelization on mjwarp [b], we were able to complete most of our successful searches within 1-3 minutes. We report the search wall-clock times on an RTX 4090, along with other metrics, in the following section on the expanded benchmark.

  2. Given the wall-clock efficiency, we surmise that experimenting with different types of objective functions (physical effect scores) and regularization should be a smooth, well-spent effort. This makes the pipeline highly practical, as the underlying policy doesn’t require retraining either.

  3. As noted in the first point, we were also able to easily transfer LJS to other steerable policies and embodiments in simulation, and the method is thus general enough to be integrated with several latent- or goal-conditioned pretrained policies.

  4. With the increasingly strong physical reasoning capabilities of LLMs and VLMs nowadays, the ease of defining per-stage effects has increased substantially, and our method of steering a low-level general controller using the Jacobian pullback to search for and satisfy these effects robustly would remain relevant.

W2: Expanded benchmark and the “complex” claim

Owing to the narrow empirical evaluation in our submitted paper, which other reviewers had also noted, we expanded the evaluation to 10 seeds across 3 humanoid tasks and 4 dextrous-hand assembly tasks. We report the results for those here.

We also conducted additional ablations with randomized starts on the shutter task and with fixed physical effect scores, without further changes. The search ablations reported in the paper have also been expanded to 10 seeds on all three humanoid tasks. We report these results here as well.

AuthorsOpenReview ↗

Response to Reviewer oBf8 (2/5)

Table: Humanoid tasks with BFM-Zero, ten new seeds (paper Table 1). Each entry gives the mean number of physical effects completed ± standard deviation, with the number of fully completed tasks in parentheses.

MethodHurdle (10 effects)Shutter (5 effects)Door + cube (6 effects)
CEM-LJS6.20 ± 3.43 (3/10)3.90 ± 2.08 (7/10)6.00 ± 0.00 (10/10)
CEM direct-z4.60 ± 2.99 (1/10)0.10 ± 0.32 (0/10)4.90 ± 1.45 (6/10)
CEM residual-BFM5.30 ± 3.92 (3/10)1.50 ± 1.08 (0/10)1.70 ± 1.25 (0/10)
Mixed-Gaussian LJS3.60 ± 3.53 (1/10)0.10 ± 0.32 (0/10)4.10 ± 2.47 (5/10)

Table: Wall-clock search time of the runs that completed the task (paper Appendix F). Mean, with the fastest run in parentheses, measured on one RTX 4090 with one search running at a time. A dash means the method never completed that task.

MethodHurdleShutterDoor + cube
CEM-LJS137 s (104 s)77 s (54 s)61 s (55 s)
CEM direct-z152 s (152 s)–72 s (58 s)
CEM residual-BFM89 s (80 s)––
Mixed-Gaussian LJS189 s (189 s)–82 s (62 s)

Table: Assembly tasks with the frozen Play2Perfect policy, ten seeds (paper Table 2, Appendix K). The policy is used as pretrained, without any fine-tuning. The first two columns give the number of fully completed tasks, and the last two give the search time of the completed runs, mean with the fastest in parentheses.

TaskCEM-LJSCEM direct-goalSearch time, CEM-LJSSearch time, CEM direct-goal
Beam assembly, first part10/1010/1062 s (29 s)46 s (29 s)
Beam assembly, second part7/104/1072 s (43 s)65 s (48 s)
Screwing9/103/1093 s (52 s)132 s (88 s)
Insertion5/104/1083 s (69 s)64 s (22 s)
All31/4021/40

Table: Shutter from ten moved and turned starting positions (paper Appendix I). Each start is moved by 0.05–0.19 m and turned by −11.2° to +11.8° from the default start, with the effect definitions unchanged. Search times are for the completed runs, mean with the fastest in parentheses.

MethodShutters completedSearch time
CEM-LJS8/1084 s (57 s)
CEM direct-z1/10158 s (158 s)
CEM residual-BFM0/10–
Mixed-Gaussian LJS2/1095 s (93 s)

Table: Search ablations on all three humanoid tasks, ten seeds each (paper Appendix G). Each entry gives the number of fully completed tasks, followed by the mean number of effects completed. The first six rows propose joint-action changes through the Jacobian (LJS), and the last five search in the latent space directly.

SearchShutter (5 effects)Hurdle (10 effects)Door + cube (6 effects)
CEM-LJS7/10, 3.93/10, 6.210/10, 6.0
MPPI [g] LJS4/10, 2.02/10, 6.39/10, 5.4
LATTICE-style [h] LJS3/10, 1.51/10, 3.67/10, 4.5
Zero-mean refinement LJS2/10, 1.01/10, 4.42/10, 2.1
Targeted-mean refinement LJS1/10, 0.50/10, 3.64/10, 4.2
Gaussian-only LJS0/10, 0.11/10, 5.52/10, 3.0
CEM direct-z0/10, 0.11/10, 4.66/10, 4.9
MPPI [g] direct-z3/10, 1.50/10, 2.36/10, 4.8
Gaussian direct-z0/10, 0.00/10, 2.74/10, 4.4
Targeted refinement direct-z0/10, 0.00/10, 1.63/10, 3.6
Random 29-D latent subspace0/10, 0.00/10, 0.80/10, 1.9

We also add a new sanity-check task with a very minimal physical-effect definition: carrying a 3 kg box through 10 points in 3D space, with only object goals and a constraint that the box never touches the floor. The same definition is reused unchanged across three generated paths, and CEM-LJS completes 12/15 runs, while the best baseline completes 3/15. This also shows that, for new tasks, the physical effect engineering can be as simple as a few object goals. (Please refer to Table 3 and Appendix J of the updated paper or the website linked for more details on this if needed).

AuthorsOpenReview ↗

Response to Reviewer oBf8 (3/5)

Regarding the claim of “Complex Contact-Rich Behaviors”, we follow the usage of prior work on humanoid control, which describes contact-rich loco-manipulation and humanoid-object interaction skills of a similar kind as complex [c, d, e, f]. In addition, the tasks now span several distinct contact regimes: whole-body support and climbing (hurdle), lifting an articulated object and passing beneath it (shutter), non-prehensile door opening while carrying an object (door-cube), tight insertion, assembly and screwing with a dexterous hand (Play2Perfect), and carrying a held object through 3D space, all elicited from frozen controllers without any retraining. That being said, following the reviewer's concern, we have removed "complex" from the title, and if the reviewer still finds the term unsuitable in other areas of the paper's text, we are happy to also remove it from the abstract and introduction.

Further, for all practical purposes, we also isolate the reasons CEM-LJS sometimes fails and correct them by running a longer search that recovers from bad “basins” or local optima it settles into. This raises the overall success rate of CEM-LJS to near-100%, while all other baselines using the same method remain underperforming. This thus reduces the trade-off for our method to only between robustness and wall-clock time, while largely removing a factor of luck.

Table: Search with retries and randomized physics, ten new seeds per task (paper Appendix I). If an effect fails, the search tries it again, and if it fails again, it goes back and searches the previous effect again. Every candidate is scored under four randomly drawn physics settings (friction, link masses, PD gains and control delay), and each run is limited to 15 minutes. Entries give the number of fully completed tasks, and the last column gives the mean search time of CEM-LJS.

TaskCEM-LJSCEM direct-zCEM residual-BFMMixed-Gaussian LJSMean search time, CEM-LJS
Hurdle (10 effects)10/106/107/102/10479 s
Shutter (5 effects)10/106/100/103/10328 s
Door + cube (6 effects)10/1010/102/1010/10145 s
All30/3022/309/3015/30

W1, Q2: Motion quality and the suggested cost terms

Since we had enough time to incorporate this suggestion as well, we did two things: (1) observe the motion quality metrics and analyze them across the methods for the existing set of search rollouts, (2) as per the final suggestion you provided, we also did a separately modified set of searches on the hurdle task (with a quick glance at the videos, the most aggressive motions are seen here). For this, instead of letting a downstream RL or learning method correct and regularize these motions, we solve the problem at its roots by adding a human-likeness regularization term to the search’s physical effect scores. We note that this is only one of the many ways to adapt our method to improve its sim2real robustness.

(1) We show, with the following metrics, how CEM-LJS in general is a much more physically agreeable search method than searching in direct joint-action space.

Table: Motion quality of CEM-LJS and CEM residual-BFM (paper Appendix H). We compare the 65 effects that both methods completed on the same task and seed, using medians and a two-sided Wilcoxon signed-rank test. The limits are those of the Unitree G1's joints and motors. Human-likeness is the score of BFM-Zero's own motion discriminator, which neither method optimizes.

Measure (per effect)CEM-LJSCEM residual-BFMEffects where CEM-LJS is betterp-value
Mechanical power (W/kg)1.592.6156/656e-07
Joint jerk, RMS (rad/s³)782166054/651e-07
Peak commanded torque / motor limit1.282.5354/654e-08
Time at the motor torque limit0.060.2043/659e-03
Joint-target rate, RMS (rad/s)2.234.0355/651e-07
Peak base angular speed (rad/s)2.934.0558/653e-10
Peak joint speed / G1 limit0.330.5054/655e-08
Excursion beyond G1 joint range (rad)0.0050.02555/651e-09
Foot slip under load (m/s)0.0200.11255/652e-08
Time with self-contact0.240.4244/652e-04
Deepest interpenetration (m)0.0080.01141/652e-02
Peak contact force (body weights)3.033.4735/650.3
Human-likeness (higher is more human)−0.89−2.4057/655e-10
Reviewer_3fNJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to search for out-of-distribution, contact-rich behaviors from a frozen latent-conditioned humanoid controller. LJS proposes task-relevant perturbations in joint-action space, maps them into latent commands through a damped inverse of the frozen policy Jacobian, and evaluates candidates with CEM and simulator rollouts. Tasks are specified as human-defined sequences of physical effects rather than complete reference trajectories, so no task-specific policy training is required. On the BFM-Zero G1 controller, LJS performs better than direct latent search, residual-BFM, and several optimizer ablations on hurdle traversal, shutter traversal, and opening a door while carrying a cube. For example, CEM-LJS completes all four shutter effects in each of the three reported seeds.

Critical strengths
  1. The combination of the two search spaces is conceptually natural. Joint-action proposals provide precise control for contact formation, while latent execution preserves whole-body coordination from the pretrained controller. The Jacobian pull-back gives this combination a clear interpretation.
  2. The baselines isolate important factors. Direct-z, residual-BFM, random latent subspaces, and several optimizer ablations indicate that the gains are not simply due to CEM, proposal dimensionality, or larger perturbations, but are associated with action-latent coupling.
Critical weaknesses
  1. The empirical evidence is limited. The main comparison uses only three seeds per method and task, while the shutter component ablation uses a single seed. Completion rates and mean differences may therefore be sensitive to initialization and CEM hyperparameters.
  2. The physical-effect definitions create a risk of benchmark adaptation. Effects, scores, and acceptance conditions were iteratively refined by the authors with LLM assistance and rollout inspection. Although the final definitions are fixed during evaluation, there is no independent task designer or blinded evaluation to test whether they favor LJS.
Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3fNJ (1/4)

We thank the reviewer for appreciating the contributions of this work and noting that the ablations are well-designed. We address the critical weaknesses mentioned by (1) increasing the breadth (4 more tasks in another embodiment) and depth (10 seeds) of empirical evidence and (2) running a controlled sanity check over tasks with a very minimal, untunable physical effect definition that was designed without observing any rollouts. Our conclusions that, due to action-latent coupled exploration, CEM-LJS outperforms and is more robust than all other baselines still hold. We hope that both of these strengthen our claim and address the reviewer’s concerns satisfactorily.

To make it easy for reviewers to access all videos and results in one place, we also link to the anonymous LJS website: https://latent-jacobian-steer.github.io/. Following the ICLR guidelines, the website is fully anonymous and does not track its visitors.

W1: More empirical evidence, along with wall-clock search-time metrics

We now report 10 seeds per task, along with the search ablations on all three humanoid tasks. CEM-LJS consistently outperforms all baselines in mean effect scores across all humanoid tasks. For this, we used completely unseen seeds and kept the physical effects fixed (note that we added a final, stable standing effect to the shutter task, bringing the total to 5 effects). The other two tasks have 10 effects (hurdle) and 6 effects (door-cube). To avoid any method-specific tuning, we also kept a fixed pair of simulation and execution horizons (40 and 20 control steps, respectively). We kept the search budgets and horizons fixed across methods and seeds, and ensured that the number of search decisions allowed for all effects is at most 30.

Table: Humanoid tasks with BFM-Zero, ten new seeds (paper Table 1). Each entry gives the mean number of physical effects completed ± standard deviation, with the number of fully completed tasks in parentheses.

MethodHurdle (10 effects)Shutter (5 effects)Door + cube (6 effects)
CEM-LJS6.20 ± 3.43 (3/10)3.90 ± 2.08 (7/10)6.00 ± 0.00 (10/10)
CEM direct-z4.60 ± 2.99 (1/10)0.10 ± 0.32 (0/10)4.90 ± 1.45 (6/10)
CEM residual-BFM5.30 ± 3.92 (3/10)1.50 ± 1.08 (0/10)1.70 ± 1.25 (0/10)
Mixed-Gaussian LJS3.60 ± 3.53 (1/10)0.10 ± 0.32 (0/10)4.10 ± 2.47 (5/10)

Table: Wall-clock search time of the runs that completed the task (paper Appendix F). Mean, with the fastest run in parentheses, measured on one RTX 4090 with one search running at a time. A dash means the method never completed that task.

MethodHurdleShutterDoor + cube
CEM-LJS137 s (104 s)77 s (54 s)61 s (55 s)
CEM direct-z152 s (152 s)–72 s (58 s)
CEM residual-BFM89 s (80 s)––
Mixed-Gaussian LJS189 s (189 s)–82 s (62 s)

Table: Search ablations on all three humanoid tasks, ten seeds each (paper Appendix G). Each entry gives the number of fully completed tasks, followed by the mean number of effects completed. The first six rows propose joint-action changes through the Jacobian (LJS), and the last five search in the latent space directly.

SearchShutter (5 effects)Hurdle (10 effects)Door + cube (6 effects)
CEM-LJS7/10, 3.93/10, 6.210/10, 6.0
MPPI [e] LJS4/10, 2.02/10, 6.39/10, 5.4
LATTICE-style [f] LJS3/10, 1.51/10, 3.67/10, 4.5
Zero-mean refinement LJS2/10, 1.01/10, 4.42/10, 2.1
Targeted-mean refinement LJS1/10, 0.50/10, 3.64/10, 4.2
Gaussian-only LJS0/10, 0.11/10, 5.52/10, 3.0
CEM direct-z0/10, 0.11/10, 4.66/10, 4.9
MPPI [e] direct-z3/10, 1.50/10, 2.36/10, 4.8
Gaussian direct-z0/10, 0.00/10, 2.74/10, 4.4
Targeted refinement direct-z0/10, 0.00/10, 1.63/10, 3.6
Random 29-D latent subspace0/10, 0.00/10, 0.80/10, 1.9
AuthorsOpenReview ↗

Response to Reviewer 3fNJ (2/4)

To understand what makes the action-latent coupling work, we also ran two further ablations on all three humanoid tasks, with ten seeds each (paper Appendix G.1). First, we rotate the joint-action proposal axes of CEM-LJS by a fixed random rotation. This barely changes the results (17/30 full tasks against 20/30 for CEM-LJS), which tells us that CEM-LJS does not depend on searching along the joint axes themselves. Second, we search along the Jacobian's own latent directions, but without the rescaling that LJS applies through the damped inverse. This completes only 9/30 tasks (Fisher's exact test, p = 0.009), although it still does far better than a random 29-dimensional latent subspace, which completes none. It is therefore both the Jacobian's directions and the rescaling of the damped inverse that make LJS work, as we argue in Appendix E.

Since the reviewer was also concerned about sensitivity to the CEM hyperparameters, we reran the main methods on the shutter task with their proposal settings changed (paper Appendix G.2). We scaled the latent proposals of CEM direct-z by 0.3 and 3 times, and the residual proposals of CEM residual-BFM by half and twice. Neither baseline gets through the shutter at any of these settings. CEM-LJS, on the other hand, completes the shutter at every setting we tried (between 2/10 and 7/10 runs), and does best at the setting used in the paper and with the damping quartered.

To test robustness against initial configurations, with the effect definitions fixed, we also start the shutter task from ten moved and turned positions: CEM-LJS completes 8/10, while the best baseline completes 2/10. (Please refer to Appendix I of the updated paper or the website linked for more details on this if needed).

We also demonstrate the generality of the method by integrating CEM-LJS with another general pretrained controller on the Sharpa-Franka setup used in Play2Perfect [a]. The steered policy here is the Play2Perfect pretrained 6-dimensional goal-conditioned policy - observe that the goal (g) can now be thought of loosely as a search variable to sequence/compose and steer the policy over the whole episode. Our baseline in this case is the direct-goal search with CEM on the Play2Perfect policy. We report results for 10 seeds across four out-of-distribution (OOD) assembly tasks. The Play2Perfect authors had to fine-tune the policy task-specifically to handle these exact four OOD tasks. However, we do not fine-tune or re-train and only use the pretrained controller for searching and eliciting the assembly behaviors. As demonstrated, CEM-LJS performs the best in this scenario as well.

Table: Assembly tasks with the frozen Play2Perfect policy, ten seeds (paper Table 2, Appendix K). The policy is used as pretrained, without any fine-tuning. The first two columns give the number of fully completed tasks, and the last two give the search time of the completed runs, mean with the fastest in parentheses.

TaskCEM-LJSCEM direct-goalSearch time, CEM-LJSSearch time, CEM direct-goal
Beam assembly, first part10/1010/1062 s (29 s)46 s (29 s)
Beam assembly, second part7/104/1072 s (43 s)65 s (48 s)
Screwing9/103/1093 s (52 s)132 s (88 s)
Insertion5/104/1083 s (69 s)64 s (22 s)
All31/4021/40

Finally, to demonstrate the ease of use of our method on any steerable latent- or goal-conditioned pretrained controller, we also show its transfer to FB-MEBE [b], ASE [c], and SPiRL [d] policies, and also run BFM-Zero [g] with LJS in a second simulator, Isaac Sim (picking, carrying, and placing a box). We don’t make any comparisons for this in the interest of time, but we expect most baseline methods will also work here, since these OOD tasks are very simple and are just showcased as a demo. The rollouts from these can be viewed on our publicly available, anonymous website, and more details are provided in our updated Appendix L.

Further, for all practical purposes, we also isolate the reasons CEM-LJS sometimes fails and correct them by running a longer search that recovers from bad “basins” or local optima it settles into. This raises the overall success rate of CEM-LJS to near-100%, while all other baselines using the same method remain underperforming. This thus reduces the trade-off for our method to only between robustness and wall-clock time, while largely removing a factor of luck.

AuthorsOpenReview ↗

Response to Reviewer 3fNJ (3/4)

Table: Search with retries and randomized physics, ten new seeds per task (paper Appendix I). If an effect fails, the search tries it again, and if it fails again, it goes back and searches the previous effect again. Every candidate is scored under four randomly drawn physics settings (friction, link masses, PD gains and control delay), and each run is limited to 15 minutes. Entries give the number of fully completed tasks, and the last column gives the mean search time of CEM-LJS.

TaskCEM-LJSCEM direct-zCEM residual-BFMMixed-Gaussian LJSMean search time, CEM-LJS
Hurdle (10 effects)10/106/107/102/10479 s
Shutter (5 effects)10/106/100/103/10328 s
Door + cube (6 effects)10/1010/102/1010/10145 s
All30/3022/309/3015/30

W2: A controlled sanity check to address the benchmark adaptation risk

We note that we designed the scores on development seeds with CEM-LJS, fixed the definitions, and never used those seeds for evaluation. Every method is evaluated with the identical definitions, so the comparison is fair whatever the definitions encode. We also note that the scores are geometrical in nature and they never describe a physical behavior that only LJS can achieve. Despite our belief that this level of control is sufficient to demonstrate the efficacy of the CEM-LJS search method over the baselines, as advised by the reviewer, we still design a completely new sanity-check task and hold off on defining the physical effect scores by inspecting any rollouts. We zero-shot define the scores for a task of carrying a 3 kg box over 10 points in 3D space. The scores have two parts: (1) object goals in 3D space, (2) a constraint that stops objects from dropping to the floor (i.e., the object must never touch the floor). This is a minimal definition that imposes no further constraints on the robot embodiment or on interaction with the object, and it is fair to all search methods. CEM-LJS again outperforms all baselines by a large margin, which shows that LJS does not need detailed effect definitions to work. We present the results of this controlled evaluation as follows.

Table: Sanity check, carrying a 3 kg box through 10 points in 3D (paper Table 3, Appendix J). Every method runs five seeds on each of three generated paths, using the search with retries. A path counts as completed when the box comes within 5 cm of all 10 points in order without ever touching the floor.

MethodPaths completedMean points reached (of 10)
CEM-LJS12/159.1
CEM direct-z2/152.5
CEM residual-BFM3/154.8
Mixed-Gaussian LJS1/151.5

With this added empirical evidence and explanations, we hope that you are satisfied with the results. We thank you again for your review, which has helped make our work much stronger. Please let us know if any more questions arise about this work, and we will do our best to address them!

Changes to the paper

We have revised the paper accordingly, with all changes marked in blue. Most changes are in Section 4, which now reports the ten-seed results on the humanoid and assembly tasks (Tables 1 and 2) and the sanity check (Table 3), and briefly answers the reviewers' questions. Elsewhere in the main text, we only clarify the wording of the abstract, introduction, related work and Section 3, and update Figure 1 to the ten-seed runs. However, our claims and methodology remain the same. The appendix adds a mathematical explanation of why LJS works (Appendix E) and the details of all new experiments: the ten-seed results, statistical tests and search times (Appendix F), the search ablations and sensitivity tests (Appendix G), motion quality (Appendix H), robustness to new starts and randomized physics (Appendix I), the sanity check (Appendix J), the assembly tasks (Appendix K), and other controllers (Appendix L).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewers

We sincerely thank the reviewers and the area chair for their valuable time and effort spent in looking into our work. Below, we summarize the improvements we made to the paper, the additional evidence supporting our work, and our responses to the reviewers' concerns. This is for the ease of the program committee, including the area chair and the reviewers. We marked all changes in blue in the paper, and all videos are on our anonymous website: https://latent-jacobian-steer.github.io/. We name the relevant reviewers in parentheses wherever needed.

(1) What the reviewers valued. All three reviewers found LJS simple, well motivated, and technically sound. Proposing changes in joint-action space gives precise control, while executing corresponding changes through the latent space maintains the pretrained controller's whole-body coordination. The Jacobian pullback we proposed gives this coupling a clear interpretation (3fNJ, oBf8, sHhd). Reviewers also agreed that our baselines and ablations sufficiently isolate this coupling's advantage (3fNJ, sHhd).

(2) More evidence (3fNJ, oBf8, sHhd). Their main shared concern was the amount of evidence for our method, and we have expanded it substantially.

  • We ran ten new seeds on all three humanoid tasks, with the same definitions, budgets, and horizons for every method. CEM-LJS fully completes 20 of 30 runs, while the best baseline completes 7.
  • All search ablations now use ten seeds on all three tasks. Overall, they show that all proposed components of LJS are required. CEM-LJS also completes the shutter task at every proposal setting we tried, while the baselines fail at all of them. This shows that the baselines are not unfairly undertuned.
  • LJS also works beyond humanoids. With the frozen Play2Perfect dexterous-hand policy, it completes 31 of 40 runs on out-of-distribution assembly tasks, against 21 for searching directly in goal space. We also show the same method transferring to more controllers (FB-MEBE, ASE, and SPiRL policies) and to another simulator, Isaac Sim.
  • We designed a new minimal task before seeing any rollouts: carrying a box through 10 points in 3D, defined only by object goals. This barebones definition leaves nothing to tune, so it cannot favor LJS over the baselines. Even with this definition, CEM-LJS clearly outperforms the other search baselines (3fNJ, oBf8).
  • From randomized starts with unchanged definitions, CEM-LJS completes 8 of 10 shutters, against 2 for the best baseline. With simple retries and randomized physics, it succeeds on every run of all our humanoid tasks, which no baseline does under the same framework (sHhd).

(3) Motion quality and hardware (oBf8, sHhd). Even without regularization, motions generated through CEM-LJS use less power, have less jerk, lower torque peaks, and less foot slip, and are more human-like than those of residual action search. They also stay within the G1's joint velocity limits. Following reviewer oBf8's suggestion, adding BFM-Zero's own discriminator as a cost makes the motions even smoother.

To the best of our knowledge, LJS is one of the first methods to elicit contact-rich, out-of-distribution behaviors such as hurdling, lifting a shutter, and opening a door from a frozen pretrained controller, without retraining, demonstrations, or teleoperation. In our opinion, this makes LJS a great tool for practitioners, especially for generating data for dynamic humanoid locomanipulation tasks.

Updates on the discussion.

  • Reviewer sHhd has noted that our additional results address most of their concerns and that their only remaining point is real-robot execution. They are also comfortable with their score being treated as a 3 if the area chair considers the current results sufficient. Reviewer oBf8 has noted that the additional experiments made the paper stronger, with remaining points on hardware applicability and on how specification effort scales.
  • To address both, we took a new task, a Unitree G1 climbing onto a 0.75 m table and standing on it, and trained a closed-loop policy from a single CEM-LJS rollout with an off-the-shelf tracker (SuperMimic). Using only on-robot sensors, the policy climbs onto the table and stands in over 95% of 512 randomized simulated trials, and in all 48 of 48 trials in a second simulator without any retraining. Writing the task's nine physical effects, searching and training took under four hours in total. The results are in the first section of our website, where the policy can also be run live in the browser on a table of the reader's choice. We are working on the hardware demo, which will be added there if successful, and we will add these results as an appendix in the final version if accepted.

We would be glad to keep addressing the reviewers' concerns as long as the discussion window is open, and we thank everyone again for helping us make our work stronger!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.