acceptodds
Desk rejected at ICLR 2027

Inverse-Conditioned World Models: Shaping Action Responses for Visual Control

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual world models support goal-directed control by predicting the consequences of candidate actions in a learned state space. However, regularizing the latent marginal alone does not bound the sensitivity of local action recovery to latent errors. We show that even exact marginal isotropy permits arbitrarily ill-conditioned local action responses and introduce Inverse-Conditioned LeWorldModel (IC-LeWM), a joint-embedding predictive model offering two independently usable approaches to action-response learning beyond marginal regularization. Inverse co-training (ICT) supervises the encoder by recovering actions from observed trajectory pairs; action-Jacobian regularization (AJR) shapes the predictor's response to nearby actions. A linear-Gaussian analysis motivates AJR's scaled-isometry target: equal nonzero singular values minimize action-recovery risk at fixed rank and total gain. Across four visual-control domains and five control interfaces, IC-LeWM configurations using ICT or AJR achieve a mean absolute success-rate gain of 5.12% over LeWM, with equal weight per domain–interface pair. The gains averaged across domains are 4.75% for CEM planning and 3.83% for direct control with freshly trained inverse heads. Geometric diagnostics further show that improved predictor conditioning alone does not ensure better control, so response shaping must also preserve predictive utility.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, visual control, representation learning, inverse dynamics, Jacobian regularization, model-based planning, Joint Embedding Predictive Architecture

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ssG6OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Inverse-Conditioned LeWorldModel (IC-LeWM), which adds action-response learning to LeWM beyond marginal regularization. The paper shows that even exact marginal isotropy permits arbitrarily ill-conditioned local action responses, and introduces two independently usable training signals to address this. Inverse co-training (ICT) supervises the encoder by recovering actions from observed trajectory pairs, and action-Jacobian regularization (AJR) shapes the predictor's response to nearby actions toward a scaled isometry. Across four visual-control domains and five control interfaces, IC-LeWM configurations using ICT or AJR achieve a mean absolute success-rate gain of 5.12% over LeWM, while geometric diagnostics show that improved predictor conditioning alone does not ensure better control.

Critical strengths
  1. The paper makes a theoretical contribution by proving that an isotropic latent distribution does not guarantee well-conditioned action responses, identifying a gap in marginal regularization such as SIGReg.

  2. Controlled studies and geometric diagnostics show that improved predictor conditioning alone does not ensure better control, providing practical guidance for future JEPA world models for control.

Critical weaknesses
  1. The reported gains depend on using a different training signal for each domain (ICT on Push-T/Reacher, AJR-FD on Two-Room/Cube) instead of one fixed objective. The main text does not explain how each signal was chosen, and the selected ICT configuration lowers success for most search planners on Reacher (Table 1). It is therefore unclear whether the gains would remain without choosing per domain.

  2. The theory behind AJR is not clearly linked to its control gains. The theory favors balanced action responses, but better predictor conditioning does not consistently correspond to better control (Figure 3b), and the best-performing configuration in the Push-T ablation (ICT+AJR-FD) has worse predictor conditioning than the unshaped baseline (Section 5.4). It therefore remains unclear whether the gains come from the proposed geometric mechanism or from other effects of the training objective.

Questions for authors
  1. How were the ICT and AJR-FD choices in the main results (Table 1) determined? Were these choices made independently of the reported evaluation results?

  2. Can you report results for each fixed choice (ICT only, AJR-FD only, and both) on all four domains and all five control interfaces under the same protocol as the main results? Gains that remain under one fixed choice would show that the method works without domain-specific selection.

  3. On Two-Room and Cube, where AJR-FD is used and the gains appear under search planners, does AJR-FD improve predictor conditioning over the unshaped baseline, and is that improvement linked to higher control success? Such evidence would help establish whether the proposed geometric mechanism contributes to the observed gains.

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for focusing on the component choices in the main table, the different planner results on Reacher, and the connection between AJR gains and the theoretical analysis. We respond to these two questions below.

1. Component selection in the main table reflects task adaptation rather than a weakness of the method

The component choice in Table 1 is not a weakness of the method. It is intended to show that ICT and AJR-FD are sensitive to task structure in different ways. AJR is more suitable for tasks that require a planner to distinguish the consequences of multiple candidate trajectories through the predictor, such as the search-planning settings in Two-Room and Cube. ICT is more suitable for direct action readout or for tasks that require recovering the first action from a distant goal, such as Reacher. We report the higher-scoring component from the two independent training paths in each domain, so Table 1 reflects task adaptation rather than a requirement to bind the two components together.

The fixed CEM results support this task-level interpretation. ICT is more favorable on Push-T and Reacher, while AJR-FD is more favorable on Two-Room and Cube. The former depends more on readable action information in the encoder, while the latter depends more on the predictor distinguishing candidate-trajectory consequences.

The decreases of the three planners on Reacher also arise from how they read the predictor. iCEM reuses the elite distribution across iterations, MPPI applies exponential weighting to trajectory costs, and Gradient planning repeatedly uses local predictor derivatives over a long horizon. These mechanisms are therefore more sensitive to the candidate ranking, response scale, and local gradients changed by ICT. CEM and GC-IDM read the learned components differently, which explains the adaptation pattern across the five interfaces.

2. AJR gains come from improving local response shape

AJR improves the predictor's local response shape and thereby helps CEM distinguish candidate actions. AJR-FD fixes the latent history and action prefix, perturbs only the current action block, and makes the predicted latent displacement closer to the action displacement at an adaptive scale, reducing response compression in some action directions. Figure 4 added in Appendix D.6 of the revision provides a direct visualization of this effect.

Using AJR requires considering both local response shape and prediction fidelity. The condition ratio alone is not a sufficient indicator of control gains. In the matched Two-Room component experiment, the condition ratio decreases from 3133.45 to 2126.42 while CEM success increases from 90% to 96%. The Push-T shape-gain comparison shows that when the condition ratio approaches 1 but prediction MSE increases from 0.00569 to 0.23240, CEM success decreases from 96% to 12%. AJR should therefore be used with predictor-based planners while preserving the predictor's effective description of the task dynamics.

Fixed CEM results across domains

DomainNo shapingICTAJR-FD
Push-T949670
Two-Room909296
Cube767678
Reacher849290

Reacher results across control interfaces

Control interfaceNo shapingICTAJR-FD
CEM829288
iCEM928474
MPPI686470
Gradient787668
GC-IDM425030

Thank you for your time and thoughtful feedback. Your comments have been invaluable in helping us improve the quality and clarity of our paper. As the discussion period will close in three days, we would greatly appreciate any further feedback on whether our response has adequately addressed your concerns. We would be happy to provide additional clarification if needed.

Reviewer_E5aBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study how to improve action selection for LeWM through inverse co-training (ICT) and action-Jacobian regularization (AJR). ICT jointly trains the encoder with an auxiliary goal-conditioned inverse-dynamics head that predicts the first recorded action between current and future observations from their latent representations. The authors also show that the sketched isotropic-Gaussian regularizer used for LeWM training does not prevent ill-conditioned action-response geometry, and propose AJR to regularize the jacobian of the map from action to predicted latent. Experiments across four visual-control domains and five controllers show modest average gains in goal-reaching success rate, with benefits depending on the task and controller.

Critical strengths
  1. The theoretical analysis exposes a limitation of the SIGReg regularization of LeWM. Propositions 1 and 2 establish that a Gaussian latent marginal does not prevent poorly conditioned action responses and explains when balanced responses reduce inverse-prediction error.

  2. The experiments show some improvements in goal-reaching success for both planning and direct control. In the main results (Table 1), AJR improves CEM success from 87% to 96% on Two-Room and from 74% to 80% on OGBench-Cube, while ICT improves direct control success from 42% to 51% on REacher. The separate Push-T ablation in Table 5 shows that ICT, AJR, and their combination respectively improve direct control success rate by 10, 8, and 14 percentage points.

Critical weaknesses
  1. I am not convinced that improved conditioning explains the reported improvements in goal-reaching success. In the Push-T ablation in Table 13, AJR-FD improves direct control success from 74% to 82% despite worsening the reported predictor condition ratio from 510 to 1221. Also, adding a gain floor to AJR-Gram improves the condition ratio from 10,655 to 1.02, but leaves direct control success and action prediction accuracy unchanged. Although the authors acknowledge this lack of correspondence in Section 5.4, the experiments leave unclear which changes induced by AJR actually produce its gains. Since the central theoretical contribution is to show that LeWM's regularizer can permit ill-conditioned action responses that may hinder inverse prediction, the lack of evidence that improved conditioning explains AJR's gains weakens the connection between the problem identified by the theory and the proposed solution.

  2. The added value relative to closely related inverse-dynamics methods needs further clarification. Existing work [1] already trains a goal-conditioned inverse head on frozen LeWM features, but the manuscript does not clarify whether its unshaped LeWM + GC-IDM baseline corresponds to that method. Furthermore, there is limited evidence for ICT's advantage over SMWM, which also uses inverse-dynamics supervision to train representations. In Table 16, ICT matches the adapted SMWM baseline on Push-T (78%) but performs substantially worse on Reacher (44% vs. 92%).

[1] Nguyen, Hoang, Xiaohao Xu, and Xiaonan Huang. "Latent Geometry Beyond Search: Amortizing Planning in World Models." arXiv preprint arXiv:2605.08732 (2026).

Questions for authors
  1. I would like to see a solid connection between the theoretical motivation and the empirical performance improvement from AJR. What about AJR explains the large goal reaching success improvements or the lack thereof in the experiments?

  2. Does the unshaped LeWM + GC-IDM baseline correspond to [1]? If not, what are the relevant differences, and how does AJR compare against [1]? Also, to what factors can I attribute the substantially worse performance of ICT on Reacher against SMWM?

I would be open to raising my score if the authors convincingly address the empirical justification for AJR and clarify ICT's contribution relative to closely related methods.

1 comment
AuthorsOpenReview ↗

Author Response

Thank you for the detailed review. We respond to the questions about the source of AJR gains, the relationship between LeWM + GC-IDM and Nguyen, and the difference between ICT and SMWM on Reacher as follows:

1. AJR gains come from improving the local response actually read by the planner

AJR-FD fixes the latent history and action prefix and perturbs only the current action block. Its theoretical motivation is that, under fixed rank, total gain, isotropic action priors, and noise, a balanced response spectrum with equal nonzero singular values reduces inverse risk. AJR-FD improves the predictor's local action response and implements this theoretical preference on the action-conditioned predictor. Once compressed action directions produce clearer latent displacements, CEM can distinguish terminal latent costs more easily, and the candidate-distribution update receives a more useful ranking signal.

These results show that the key to AJR's control gains is improving the predictor's local response shape and providing CEM with a clearer candidate-ranking signal, rather than merely reducing the condition ratio. In Two-Room, the condition ratio decreases from 3133.45 to 2126.42 while CEM success increases from 90% to 96%. However, the Push-T shape-gain comparison shows that when the condition ratio approaches 1 but prediction MSE increases from 0.00569 to 0.23240, CEM success drops from 96% to 12%. AJR should therefore be applied to predictor-based planners while preserving prediction fidelity.

For GC-IDM, AJR gains come from the effect of joint training on the encoder's action representation because GC-IDM does not call the predictor at deployment. In Push-T, AJR-FD increases GC-IDM success from 74% to 82% even though the predictor's condition ratio changes from 510 to 1221. This shows that GC-IDM reads more usable action information from the encoder, so AJR affects GC-IDM through a different path than CEM.

2. Relationship between LeWM + GC-IDM and Nguyen, and ICT's task adaptation on Reacher

Nguyen's main result and our LeWM + GC-IDM score differ substantially because the experimental settings are different. Nguyen's main result trains GC-IDM on the complete Reacher dataset, whereas our Reacher GC-IDM head is trained on only 90% of the episodes, with the remaining 10% used for validation, and is then evaluated on all episodes. Thus, our head uses a stricter and more conservative training-data split, which leads to a lower score. In addition, Nguyen does not clearly report the qpos threshold used for Reacher evaluation, so it is impossible to confirm whether the success criterion matches our per-joint error threshold of 0.1 radians. However, the LeWM and our method reported in Table 1 under Reacher/GC-IDM are compared under the same configuration, so that comparison is fully fair.

ICT is worse than SMWM on Reacher mainly because the two methods supervise action information at different temporal scales. SMWM recovers the immediate action from an adjacent transition, and its adapted setting uses a larger inverse weight and omits SIGReg. ICT instead uses a distant goal and a horizon-conditioned head to regress the first action. The same distant goal can correspond to multiple first-action paths, so squared-error regression averages these actions. Reacher direct control depends on immediate joint actions, which is why adapted SMWM reaches 92% while ICT reaches 44%.

The decreases of ICT on iCEM, MPPI, and Gradient on Reacher arise from a mismatch between how these planners read predictor responses and ICT's supervision target. iCEM reuses the elite distribution across iterations, so early candidate-ranking errors affect later sampling. MPPI uses exponential trajectory-cost weights, which can amplify changes in the predicted response scale. Gradient planning repeatedly propagates local predictor derivatives over a long horizon, so local mismatch accumulates. CEM reselects candidates at each iteration, while GC-IDM reads actions directly from the encoder. ICT therefore behaves differently across control interfaces.

Reviewer_iPxcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an inverse-conditioned version of LeWM. The method combines the standard LeWM loss with an inverse-dynamics loss on trajectories, called inverse co-training (ICT) in the paper, and action-Jacobian regularization (AJR). The authors analyze and motivate the geometry targeted by AJR. Overall, the results show improvements over the LeWM baseline.

I find the proposed method and results somewhat incremental in current form, but the analysis and theoretical propositions can be much more interesting and promising than the method itself.

Critical strengths
  1. The paper explores an interesting direction that connects learned representations with actions. This is important and underexplored in current JEPA-based research. The paper is well structured and easy to read.

  2. The paper provides useful insights, supported by reasonable theoretical propositions. The overall quality of the method’s explanation is high, and the inclusion of Gram-based regularization is helpful.

  3. The paper presents useful positive and negative empirical results.

Critical weaknesses
  1. The method’s overall novelty is limited in current presentation form. ICT-based approaches are widely used in goal-conditioned world modeling, including in related methods such as [1]. Jacobian analysis and its application to training are also well established. I think deeper grounding on prior work can significantly strengthen current paper.

  2. The experimental results are somewhat weak and do not provide a unified recipe for when and how to use AJR and ICT. The main missing element is an analysis explaining how to use action conditioning effectively and scalably.

[1] - Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models

Questions for authors

I ask the authors to provide:

  1. A more detailed comparison with PLDM and stronger grounding in related work on Jacobian-based analysis and inverse-dynamics learning for world modeling. A textual analysis that clearly distinguishes the approaches may be sufficient; if it is not, an experimental comparison would be helpful.

  2. More comprehensive guidance on how to use AJR, including an explanation of the Jacobian-based interpretation of the action distribution. (I think some vizualizations may be helpfull).

If both questions are fully addressed, I will raise my score.

4 comments
AuthorsOpenReview ↗

Author Response

Thank you for your careful review. Our central novelty is a theory-driven decomposition of action-related representations in reconstruction-free JEPA control into latent marginal regularity, data action identifiability, and local action-response geometry. We prove that the first two do not imply the third and give an inverse-risk objective under fixed rank, gain, and noise with predictor-to-environment transfer conditions. ICT and AJR follow directly from this decomposition. ICT preserves action information in the encoder, while AJR constrains the predictor's response to candidate actions. This yields an action-response learning problem definition with explicit boundaries and two independently usable training paths not covered by prior work or their combination. Figure 1 also marks ICT and AJR as independently usable and distinguishes encoder action-information supervision, predictor-response shaping, and the GC-IDM and search-planning interfaces.

1. Point-by-point distinctions from prior work

PLDM and IC-LeWM address the same broad problem: learning an action-aware latent world model from offline data. PLDM regresses actions from adjacent latent pairs in its joint objective. ICT instead recovers the recorded first action from (z_t, z_{t+h}, h), allows the future goal to extend beyond the predictor's short window, and makes action identifiability a goal-conditioned constraint across time. AJR targets local response geometry for the fixed history and current action. ICT therefore constrains encoder-side action identifiability, while AJR constrains predictor-side local response.

Nguyen points out that distributional isotropy does not imply local near-isometry. We additionally give a Gaussian-preserving and rank-preserving construction that keeps the marginal, action information, and response rank unchanged while allowing the identifiable response condition number to become arbitrarily large. Nguyen trains a GC-IDM on frozen LeWM features for goal-conditioned, search-free control. We study world-model training, with ICT updating the encoder and AJR acting on the predictor, and connect the result to inverse risk and predictor-to-environment transfer.

SMWM and IC-LeWM both use inverse supervision, but differ in the supervised object and temporal span. SMWM uses adjacent transitions. ICT uses a distant goal, the first action, and horizon conditioning. AJR does not rely on an inverse head and directly shapes the predictor's local action response.

2. Why inverse supervision does not automatically cover AJR

Inverse MSE measures regression risk conditioned on X=(z_t,z_{t+h},h). Lower inverse error can reflect more action information in the representation or an easier inverse head, so it does not imply improved predictor-Jacobian conditioning. A distant goal can correspond to multiple first-action paths. AJR instead measures the predictor derivative under action perturbations with a fixed context. A smooth reparameterization can preserve inverse risk while worsening the response condition. Inverse recoverability and predictor local response are therefore independent, so ICT and AJR address different problems.

3. Prior Jacobian work and AJR's novelty

AJR's novelty is to adapt a local geometric constraint to a reconstruction-free JEPA action-conditioned predictor. It fixes the latent history and action prefix, perturbs only the current action block, and shapes the response that the planner compares. Existing work on isometry, Jacobian regularization, and latent control does not impose this constraint on the same predictor or provide this problem decomposition. AJR-FD uses two predictor evaluations and a shared EMA scale, making it suitable for attention predictors without higher-order differentiation through the full Jacobian. It balances local response gains across action directions. Under fixed rank, total gain, isotropic action prior, and noise, equal nonzero singular values minimize inverse risk, motivating the objective.

4. How to use AJR

At the task level, AJR is most suitable for goal-reaching tasks that evaluate multi-step action consequences through imagined rollouts, especially when unequal latent responses make candidate trajectories difficult to distinguish. The Two-Room and Cube search tasks have this property. If a task mainly uses GC-IDM for direct action readout, or if a distant goal has multiple possible first-action paths, preserving readable action information in the encoder is more direct, so ICT is preferable.

Use AJR when the predictor's local response becomes a control bottleneck, and preserve prediction fidelity while shaping it. With fixed latent history and action prefix, nearby perturbations satisfy Δz^≈B^t(δ−δ′)\Delta \hat z \approx \hat B_t(\delta-\delta'). Thus compressed directions produce similar predicted outcomes, making candidate actions easier for CEM to rank, with success increasing from 90% to 96%. Figure 4 in Appendix D.6 also visualizes how AJR works.

Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of our paper. With the discussion period closing in three days, we would appreciate any further feedback on whether our response addresses your concerns. We would be happy to provide further clarification.

Thank for clarifications. I will raise my score.

We sincerely appreciate your thoughtful review and your support in raising your score. Your feedback helped us clarify several important details, and we will ensure all of them are fully integrated into the revision.

Comments on the paper

2 comments

Dear reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

Dear Area Chair,

We have posted detailed point-by-point responses to Reviewers iPxc, E5aB, and ssG6 in the rebuttal thread, but have not yet received further feedback. Our responses address the reviewers’ main concerns, including the novelty of IC-LeWM relative to PLDM, SMWM, and prior Jacobian- and inverse-dynamics-based methods; the empirical connection between AJR’s response geometry and control gains; the task-dependent roles and non-additive interaction of ICT and AJR-FD. We also directly answer each reviewer’s questions and clarify the relevant experimental protocols and limitations.

With the discussion period closing on November 18 (AoE), if possible, we would be grateful if you could encourage the reviewers to take a look at our responses before the final decision.

Thank you very much for your time and coordination.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.