Latent Jacobian Steer: Eliciting Complex Contact-Rich Behaviors from Pretrained Humanoid Controllers

Given a pretrained general latent-conditioned whole-body humanoid controller, how can we elicit complex and contact-rich out-of-distribution behaviors? Joint-action exploration enables direct task-specific adjustments but can disrupt coordination learned by the pretrained controller, whereas latent-command exploration retains it but is difficult to steer toward desired physical effects. To combine the complementary strengths of these exploration spaces, we propose Latent Jacobian Steer (LJS), which couples sampling in direct joint-action space with the semantics of the learned frozen controller's input latent space. For a task specified through desired physical effects, we sample task-relevant joint-action perturbations and map them through a damped local inverse of the policy Jacobian to obtain candidate latent commands. These commands are executed through the frozen pretrained policy, while exploration is directed toward the physical outcomes required by the new task. Candidate outcomes are evaluated in simulation and composed into latent-command sequences that accomplish the task. Ablations show that neither action-space nor latent-space exploration alone provides the same capability as their coupling. Across diverse out-of-distribution tasks, test-time searching using this method produces contact-rich behaviors, including hurdling, manipulating articulated objects, pivoting large objects against the environment, and opening doors through non-prehensile contact.