acceptodds
Under review as a conference paper at ICLR 2027

Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control.

Abstract

World models jointly learn latent representations and dynamics that predict how high-dimensional observations evolve under actions. In this work, we propose a JEPA-style world model in which, rather than learning arbitrary latent dynamics, we restrict them to follow a bilinear parameterization. This structure enables efficient planning and control while shifting the modeling burden onto the encoder, encouraging richer representations that expose the controllable geometry of the system. In particular, this structured parameterization allows us to structurally enforce action recoverability, thereby preventing representation collapse by construction. Although prescribing a bilinear parametrization may appear restrictive, we show that a broad class of nonlinear dynamical systems admits a transformation under which the dynamics become bilinear. Empirically, we show across standard 2D and 3D control tasks that representations with bilinear-parameterized dynamics can be learned directly from high-dimensional observations, reducing planning time by nearly three orders of magnitude while retaining or even improving control accuracy. We also propose more demanding regimes of longer-horizon planning and real-time control, and demonstrate that our method succeeds in both, moving JEPA-style world models beyond short-horizon offline planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World models, JEPA, representation learning, latent dynamics, model-based control, bilinear systems, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_TotmOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a Bilinear World Model that imposes structure on the latent dynamics. Instead of learning a representation and a forward model using separate neural models, which can lead to representation collapse, the proposed method is constructed in a way where the learned state representation must adhere to the specified dynamics. Given such a method, an action can be recovered directly from the latent transition. The authors show that this Bilinear World Model reduces planning time by three orders of magnitude while matching or improving over current SOTA World Model approaches.

This work takes a novel and promising direction. I recommend accepting it.

Critical strengths
  • The idea is clever and effective. Restricting the dynamics model and forcing structure into the latent representation is timely and creative. The results show that this is a promising direction to improve World Model representation and planning.

  • The authors compare against SOTA World Models (SMWM and LeWM). They show that the Bilinear model matches or surpasses performance while significantly reducing planning time.

Critical weaknesses
  • A conclusion and, especially, a limitations section are missing completely. This needs to be addressed.
Questions for authors
  1. What's still a bit unclear to me is the source of the speedup. In the main text, the source is unclear, while in the appendix it reveals a bit that the speedup is not due to the proposed bilinear model, but a lot of it seems to come from the Gauss-Newton planner itself and related tricks like the reduced action space mentioned in B.2.3. It would be helpful to be more explicit in the main text.
1 comment

We sincerely thank the reviewer for their encouraging assessment of our work. We particularly appreciate their recognition of the proposed structured dynamics as a promising direction for improving world-model representations and planning efficiency. We agree that the manuscript would benefit from a more explicit discussion of its limitations and a clearer explanation of the source of the computational speedup. We address both points below.


1. Conclusions and limitations

We agree that adding a conclusion and stating clearly the limitations is necessary. We combined the requests into a Conclusion and Limitations section, which reads:

We introduced Bilinear World Models, a JEPA-style framework that imposes structured latent dynamics to encourage representations that leverage the controllable structure of the system while preventing representation collapse through action recoverability. We established the expressiveness of bilinear embeddings through Koopman theory and showed that their structure enables efficient planning using Gauss–Newton optimization. Across four benchmark environments, our method matches or improves the performance of state-of-the-art JEPA-style world models while reducing planning time by approximately three orders of magnitude, enabling longer-horizon planning and real-time control.

Limitations and future work. Our experiments focus on the standard world-model benchmark suite, which includes nonlinear dynamics, contact-rich interactions, and 2D and 3D manipulation, but remains limited in environmental complexity. Our guiding principle is to impose the simplest possible latent dynamics for which a suitable transformation of the original system exists. While bilinear-parameterized dynamics proved sufficiently expressive across the evaluated tasks, more complex environments may require richer dynamics. A natural direction for future work is therefore to explore more expressive structured parameterizations, preserving the benefits of structured dynamics for representation learning and efficient control while extending the range of systems that can be represented. More concisely, simpler dynamics enable more efficient control, but must remain sufficiently expressive to capture the underlying system. In the standard world-model evaluation suite, bilinear dynamics proved more than sufficiently expressive. However, more complex environments may require richer structured dynamics to maintain this balance between expressiveness and efficient planning, which signals an interesting line of work.

In addition, we invite the reviewer to consider the additional experiments presented in our response to reviewer 7NTT, which provide further insights into the learned representations and the limitations of our approach in more complex environments.


2. Source of Speedup

We agree with the reviewer that we should make more explicit how the bilinear dynamics enable faster planning, and appreciate the opportunity to improve the presentation of our work.

The speedup comes from using the Gauss–Newton (GN) planner instead of CEM. The proposed bilinear dynamics enable the use of GN without losing performance.

The GN can also be applied to unstructured neural-network dynamics. However, our experiments in Appendix C.2 (Figure 7) show that applying the same planner to SMWM and LeWorld results in substantially degraded control performance.

This is because GN relies on first-order information about the dynamics (i.e., derivatives). In our method, these derivatives are directly available through the matrix (M(z)) of the bilinear parameterization. For neural-network dynamics, these derivatives can be computed through automatic differentiation. However, neural networks are trained for discrete transition accuracy (i.e., predicting the next state given the current state and action), which does not guarantee that differentiating their predictions with respect to the action yields useful information for gradient-based planning, which is exactly the limitation we see in practice.

In summary, structured bilinear dynamics give access to accurate first-order information, enabling the use of planners such as GN. We hope this addresses the reviewer's question and would be happy to provide further clarification.

Reviewer_jfTEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a bilinear transition function for a jepa like model. This leads to the use of gradient based planner and leveraging gradient information in the planning leading to a more efficient planning performance in terms of wall time.

Critical strengths
  1. The transition function is explained clearly
  2. The method increases planning efficiency in wall time drastically while maintaining performance
Critical weaknesses
  1. Limitations are not communicated very clearly,
  2. The paper is missing a conclusion section.
  3. the learned latent state is not analyzed clearly in comparison with the non linear jepa transiions.
  4. How does the lipshitzness assumption affect your performance, especially in environments where this is not satisfied
Questions for authors

What are the limitations fo your approach ? are there environments where this structure would lead to performance degredation.

3 comments

We appreciate the feedback of the reviewer and hope to address the raised concerns below.


1. Conclusions and limitations

We agree that a dedicated discussion of limitations would strengthen the paper. We have added a Conclusions and Limitations section to the revised manuscript, which reads:

We introduced Bilinear World Models, a JEPA-style framework that imposes structured latent dynamics to encourage representations that leverage the controllable structure of the system while preventing representation collapse through action recoverability. We established the expressiveness of bilinear embeddings through Koopman theory and showed that their structure enables efficient planning using Gauss–Newton optimization. Across four benchmark environments, our method matches or improves the performance of state-of-the-art JEPA-style world models while reducing planning time by approximately three orders of magnitude, enabling longer-horizon planning and real-time control.

Limitations and future work. Our experiments focus on the standard world-model benchmark suite, which includes nonlinear dynamics, contact-rich interactions, and 2D and 3D manipulation, but remains limited in environmental complexity. Our guiding principle is to impose the simplest possible latent dynamics for which a suitable transformation of the original system exists. While bilinear-parameterized dynamics proved sufficiently expressive across the evaluated tasks, more complex environments may require richer dynamics. A natural direction for future work is therefore to explore more expressive structured parameterizations, preserving the benefits of structured dynamics for representation learning and efficient control while extending the range of systems that can be represented. More concisely, simpler dynamics enable more efficient control, but must remain sufficiently expressive to capture the underlying system. In the standard world-model evaluation suite, bilinear dynamics proved more than sufficiently expressive. However, more complex environments may require richer structured dynamics to maintain this balance between expressiveness and efficient planning, which signals an interesting line of work.

In addition, we invite the reviewer to consider the additional experiments presented in our response to reviewer 7NTT, which provide further insights into the learned representations and the limitations of our approach in more complex environments.


2. Latent space comparison

We agree with the reviewer that the comparison with nonlinear JEPA world models deserves further clarification.

We claim that imposing substantially more restrictive bilinear dynamics does not prevent the emergence of physically meaningful latent representations. As shown in Figure 2, the principal components of our representations align closely with physical quantities of the systems. These observations are consistent with the structures reported in previous JEPA-style world models, including LeWM and StableWM.

Nevertheless, comparing the latent structure with the other world models would add on the manuscript soundness. We analyze the latent representation by using the Disentanglement, Completeness, Informativeness (DCI), an evaluation framework for learned representations Eastwood & Williams (ICLR 2018, A Framework for the Quantitative Evaluation of Disentangled Representations).

Table 1. Disentanglement. It measures the extent to which each latent dimension encodes a single physical factor rather than multiple ones.

taskBilinear WM (ours)SMWMLeWorld
TwoRoom0.970.760.26
PushT0.370.580.14
Reacher0.190.110.25
Cube0.320.300.16

Table 2. Completeness. It measures each physical factor spread over latent dimensions.

taskBilinear WM (ours)SMWMLeWorld
TwoRoom0.480.200.08
PushT0.350.220.03
Reacher0.060.050.08
Cube0.100.100.04

Table 3. Informativeness. It measures how accurately the ground-truth physical factors can be predicted from the learned latent representations. We use linear regression.

taskBilinear WM (ours)SMWMLeWorld
TwoRoom1.000.780.99
PushT0.920.950.96
Reacher0.920.740.99
Cube0.990.990.96

We observe that Bilinear WM generally learns structured latent representations, with better separation of physical variables (disentanglement) in 2/4 tasks and more compact encoding of each variable (completeness) in 2/4 tasks, while preserving the information needed to recover physical states (informativeness), where it leads in 2/4 tasks. This demonstrates that simpler latent dynamics can retain relevant physical information while providing better-structured representations for control.

3. Lipschitz assumption and its implications

This is a very good question. The conditions of Proposition 1 (including the Lipschitz assumption) guarantee the existence of an exact bilinear representation. When Lipschitzness does not hold, this guarantee no longer applies. However, a good (but approximate) representation can still be achieved in practice.

One example of this is TwoRoom. Near the doorway separating the two rooms, a small change in vertical position determines whether a horizontal action can cross the wall or not. In the simulation, two states separated by only one pixel can produce next states 25 pixels apart (i.e., crossing the room or hitting the wall), showing a violation of the Lipschitz condition.

Table 4. To evaluate how this affects the dynamic prediction, we tested the three world models on 300 transitions concentrated around the doorway. The table reports mean one-step prediction errors in physical coordinates (pixels).

ModelDoorway-FarDoorway-NearOverall
Bilinear (ours)7.5810.219.34
SMWM5.4510.938.88
LeWorld7.6914.2911.64

These results show that discontinuous transitions are harder to predict, but do not prevent our approach from learning a useful approximation and achieving strong performance. We also remark that the three methods have a harder time predicting the transitions near a discontinuity of the dynamics, not only our method.

In conclusion, the Lipschitz assumption provides sufficient conditions for an exact bilinear representation, but is not necessary for our method to perform well in practice. In particular, Table 2 shows that discontinuous dynamics are more challenging to predict, but this difficulty affects all evaluated world models, not only our structured approach.


Additional feedback

We thank the reviewer again for their assessment and suggestions, which have helped us improve the clarity of the manuscript. We hope these revisions address the concerns raised. We would also welcome any further feedback regarding specific technical or empirical limitations that the reviewer considers important for their overall assessment, so that we can address them as directly as possible.

Thanks to the authors for addressing my concerns. I think the added results definitely increase the paper's value. I will raise my score to 3.

Reviewer_7NTTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper restricts the latent dynamics of a JEPA-style world model to a normalized bilinear form, z′=Az+M(z)az' = Az + M(z)a, and argues that this prevents representation collapse.

Modeling: They adopt the latent encoder from LeWM, in which a ViT-Tiny encoder maps each observation to a 192-dimensional latent. The encoder and the dynamics are trained jointly with two terms: (i) a forward-prediction loss, the squared error between the encoded next observation and the bilinear prediction Az+M(z)aAz + M(z)a; and (ii) an action-recovery loss, weighted by λ=30\lambda = 30, which asks that the action taken be recoverable from the latent change through the pseudo-inverse of M(z)M(z), so that distinct actions cannot produce indistinguishable latent transitions. The action map is normalized with a Cholesky-QR step, so that M(z)⊤M(z)=R⊤RM(z)^\top M(z) = R^\top R for a learned matrix RR at every state and the map stays full rank as long as RR does.

Planning: Instead of the Cross-Entropy Method (CEM) for planning, they exploit the bilinear structure to formulate a Gauss-Newton (GN) planner. CEM uses no sensitivity information and needs many sampled rollouts, whereas the bilinear dynamics stay explicitly differentiable in both states and actions; the Jacobian of the final latent with respect to the action sequence then gives a local linear model of the goal error, which GN minimizes in a few steps, making it efficient.

Experiments: They conduct experiments over 2D navigation (TwoRoom), pushing (PushT), reaching (Reacher) and 3D manipulation (OG-Cube). At goal offset 25 they show an increase in success rate of 0 to 16 percentage points over the strongest baseline (100/90/75/99% against 97/90/67/83%), as well as a reduction in planning wall-time from 7-32 minutes to 0.7-3.1 seconds per episode, compared to baselines of LeWM and SMWM planning with CEM. On moving-goal tasks the margin is about 27 percentage points on average.

Critical strengths

1. Learning structured dynamics: The paper imposes a bilinear structure on the latent dynamics, which keeps the world model simple and well-conditioned for control [Section 3].

2. Efficient controller: The bilinear structure in-turn makes a fast gradient-based (Gauss-Newton) planner possible in place of sampling-based search, so planning becomes both cheaper and more accurate [Section 5, Figure 7].

Critical weaknesses

1. Four low-dimensional goal-reaching tasks are too few to show that the method generalizes.

  • Claim: "a broad class of nonlinear dynamical systems admits a transformation under which the dynamics become bilinear" [Abstract], extended to "any action-affine nonlinear system" [p.2, (C2)]. GN is motivated by sampling becoming expensive "as the planning horizon or action dimension increases" [p.5, Section 5].
  • Evidence:
    • Every experiment uses the four tasks of the LeWM suite: TwoRoom, PushT, Reacher and OG-Cube [p.6, Section 6]. Their action dimensions are 2 to 5 [Table 1, p.16], and all four are single-agent goal-reaching from images.
    • Reacher is the only dm_control task, although dm_control is in the software stack [Appendix B, p.13].
    • The planner shrinks the problem to 10-15 unknowns with 5-step coherent action blocks [Table 1], and a learned residual is added for "contact-rich environments" [Equation 32, p.15]. Both suggest that larger action spaces and richer contact are where the bilinear restriction is under strain, and the experiments do not reach that regime.
  • Why it matters: the bilinear form trades expressiveness for structure. Four tasks on which it works do not show where that trade breaks: locomotion with frequent contact switching (Cheetah, Walker, Quadruped), high-dimensional or underactuated control (Humanoid, Dog, Acrobot), or long manipulation chains. Those are the settings that would show whether the collapse and planning benefits hold across different action and world-learning dynamics.
  • Fix: evaluate on a broad set of dm_control tasks that spans action dimension (from Cartpole to Humanoid and Dog) and contact structure, against the same baselines under both CEM and GN, and report where the bilinear model falls behind; plot success and planning time against action dimension.
Questions for authors

I don't have any specific questions. It would help if the authors could give more evidence for the performance by extending the experiments to the full dm_control suite, as suggested in Weakness 1.

1 comment

We thank the reviewer for the proposed additional experiment. We agree that evaluating more complex environments would help clarify the scope and limitations of our approach. To address this concern, we have conducted additional experiments on three of the suggested dm_control environments.


1. Additional experiments

Following the reviewer's suggestion, we extended our evaluation to Cartpole, Quadruped, and Humanoid, with action dimensions of 1, 12, and 21, respectively, spanning increasingly complex dynamics, contact-rich interactions, and higher-dimensional action spaces.

Table1. Success rate (%).

MethodCartpoleQuadrupedHumanoid
Ours39486
SMWM25426
LeWorld29155
Random1960

Table 2. Planning time (min).

MethodCartpoleQuadrupedHumanoid
Ours0.30.60.1
SMWM4.41.83.6
LeWorld24.211.923.7

Although all three world models outperform random actions, their absolute success rates remain limited on these more challenging environments, particularly Humanoid. Nonetheless, our method achieves the highest success rates on Cartpole and Quadruped, while matching SMWM on Humanoid, and consistently reduces planning time.

We further analyze the dimensionality of the learned latent representations.

Table 3. Number of Principal Components accounting for 90 % of the variance in the latent space.

MethodTwoRoomPushTReacherOG-CubeCartpoleQuadrupedHumanoid
Ours13071865685
SMWM31967489
LeWorld64786590252715
Action dim222511221

Table 3 suggests that Bilinear World Models learn higher-dimensional representations as the environment dynamics become more complex. In contrast, LeWorld maintains relatively high dimensionality across environments, consistent with its Gaussian latent regularization, while SMWM produces representations whose dimensionality is generally close to that of the action space, consistent with its action-recoverability objective.

Imposing simple bilinear dynamics on increasingly complex systems shifts the modeling burden onto the encoder, encouraging higher-dimensional representations. We hypothesize that this ability to adapt the representation's complexity to that of the underlying system contributes to the improved control performance of our method on these more challenging environments.

We truly thank the reviewer for motivating these additional experiments, which revealed an interesting property of our approach. We believe these findings strengthen our work and will incorporate them into the revised manuscript.


2. Clarification of the theoretical expressiveness claim

We clarify the claim highlighted by the reviewer:

"A broad class of nonlinear dynamical systems admits a transformation under which the dynamics become bilinear" [Abstract], "any action-affine nonlinear system" [p. 2, (C2)].

This claim follows from our theoretical result based on Koopman theory, rather than from the empirical evaluations. Specifically, Proposition 1 establishes the existence of an exact, potentially infinite-dimensional bilinear embedding for continuous-time action-affine systems satisfying the stated assumptions.

However, this result does not guarantee that the finite-dimensional bilinear representations used in practice are sufficiently expressive, nor that such representations can be learned accurately from finite data.

We will clarify this distinction in the revised manuscript. We will revise the Abstract and Introduction to distinguish the theoretical existence result from the practical expressiveness of finite-dimensional bilinear models.


3. Scope and limitations of the proposed approach

We further clarify the scope of our work, addressing the absence of a dedicated Conclusions and Limitations section noted by reviewers jfTE and Totm.

Our objective is to demonstrate that bilinear latent dynamics are sufficiently expressive across the complete LeWM benchmark suite while significantly improving planning efficiency.

However, we acknowledge that more complex environments may require richer, yet still structured, latent dynamics. In such cases, our guiding principle remains the same: to impose the simplest possible latent dynamics for which a suitable representation of the underlying system exists. Exploring more expressive structured parameterizations is a natural direction for future work, which we now discuss in the revised Conclusions and Limitations section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.