Inverse-Conditioned World Models: Shaping Action Responses for Visual Control

Visual world models support goal-directed control by predicting the consequences of candidate actions in a learned state space. However, regularizing the latent marginal alone does not bound the sensitivity of local action recovery to latent errors. We show that even exact marginal isotropy permits arbitrarily ill-conditioned local action responses and introduce Inverse-Conditioned LeWorldModel (IC-LeWM), a joint-embedding predictive model offering two independently usable approaches to action-response learning beyond marginal regularization. Inverse co-training (ICT) supervises the encoder by recovering actions from observed trajectory pairs; action-Jacobian regularization (AJR) shapes the predictor's response to nearby actions. A linear-Gaussian analysis motivates AJR's scaled-isometry target: equal nonzero singular values minimize action-recovery risk at fixed rank and total gain. Across four visual-control domains and five control interfaces, IC-LeWM configurations using ICT or AJR achieve a mean absolute success-rate gain of 5.12% over LeWM, with equal weight per domain–interface pair. The gains averaged across domains are 4.75% for CEM planning and 3.83% for direct control with freshly trained inverse heads. Geometric diagnostics further show that improved predictor conditioning alone does not ensure better control, so response shaping must also preserve predictive utility.