acceptodds
Withdrawn from ICLR 2027

MEND: Measuring and Constraining Latent Dynamics in World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models used for planning and simulation must preserve dynamics under repeated prediction. In joint-embedding predictive architectures (JEPAs), one-step accuracy does not ensure this behavior. We formulate latent prediction as an approximate semi-conjugacy. In the linear-symplectic setting, this relates dimensional collapse to encoder rank and prediction drift to predictor spectrum. MEND measures frequency, instability, and contraction in latent trajectories and uses these measurements to constrain a new predictor while keeping the encoder frozen. An orbit-averaged log-determinant separates physical dissipation from least-squares attenuation. An autonomously iterated V-JEPA 2 predictor has at least 12 positive finite-time Lyapunov exponents and loses an oscillator signal within to steps, while the encoder retains it. On a real pendulum video, MEND constrains a conformal-symplectic predictor in a two-dimensional observable. It extends the open-loop horizon from 21 to 651 steps (0.6 to 18.4 periods), approximately 31 times, while following the observed decay. These results support video-based system identification and prediction of mechanical motion using frozen representations. Extending the constraints to action-conditioned world models could support longer-horizon planning and control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architecture, long-horizon prediction, Lyapunov exponents, symplectic structure, representation collapse

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WS1nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the setting of JEPA-like world models and under which conditions they experience failure modes. The paper claims two main contributions: a theoretical distinction between representation collapse due to the encoder discarding state information and prediction drift due to JEPA only supervising next-state prediction and not full rollouts and analysis of these failure modes on a frozen V-JEPA 2 and a potential solution to address this coined MEND which instead fits the parameters of a damped oscillation model in latent space. The proposed method is experimentally verified on video observations of a mechanical pendulum.

Critical strengths
  1. The problem setting is relevant to the community. Also, the idea of adding physical priors to learned dynamics is an interesting research direction
Critical weaknesses
  1. The primary issue with the paper is that it is badly written, not well presented which makes it very hard for the reader to follow the paper - in terms of theory, methodology and results, etc. Indeed, most of the content/text seems to be heavily AI-generated. Therefore, the paper in its current form does not provide value to the reader or the broader community as the insight into the theory, the analysis of the frozen V-JEPA2 or the proposed method is non-existent/extremely limited.
  2. The validation of the proposed method seems to be limited as it has only been done on the experimental pendulum which will (naturally) agree well with the mechanical prior on the latent dynamics of a conformal-symplectic DMD.
Questions for authors
  1. What other systems beyond the mechnical pendulum do you envision this proposed conformal-symplectic DMD latent dynamics to be beneficial for? Would the latent dynamics have to be changed/adjusted?
Reviewer_rLkqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the dynamical distortion problem of JEPA world models during open-loop prediction. By formulating latent prediction as an approximate semi-conjugacy, this paper attributes dynamical distortion to the dimensional collapse of the encoder and the drift of the predictor in a linear-symplectic setting. This paper introduces MEND, a method that measures frequency, Lyapunov exponents, and contraction rates from representations generated by a frozen encoder, and uses these measurements to constrain the predictor and mitigate prediction drift. In a two-dimensional space of observables of a single real pendulum video, the authors use a conformal-symplectic predictor and report that the open-loop prediction horizon was extended from 21 steps for the V-JEPA 2 predictor to 651 steps, while making the decay trend of the predicted amplitude closer to the observed decay.

Critical strengths

S1 This paper proposes MEND, which constrains the predictor by measuring the dynamical properties of the frozen representation, thereby mitigating the drift in open-loop prediction. This use of dynamical diagnostics to guide predictor design is interpretable and allows for prediction of low-dimensional dynamics using pre-trained representations without retraining the encoder.

S2 To mitigate overestimation of dissipation rates that can arise from observation noise, the paper directly fits transitions between states separated by multiple time steps and uses the estimated contraction rate to constrain the predictor. This technique helps preserve the physical dissipation of the system while reducing excessive decay induced by noise in the observations.

Critical weaknesses

W1 The data usage and the evaluation protocol for the real-pendulum experiment remain unclear. While the paper specifies the train/test split for the angle probe, it does not explicitly state whether the same split was applied to the estimation of contraction rates, nor whether prediction was evaluated on independent test data. This makes it difficult to assess the predictive performance of the method on data not used for fitting.

W2 The predictive advantages of the proposed method over TLS-DMD and fb-DMD require further validation. The OOD rate of all three methods in Table 1 is 0%. MEND’s predicted amplitude decay is closer to the observed decay, but this comparison is based on only a single real video segment and is insufficient to establish whether the improvement holds across other trajectories. Furthermore, the existing experiments do not distinguish the respective contributions of more accurate contraction-rate estimation and conformal symplectic constraints. The paper lacks a comparison where existing DMD methods are calibrated using the contraction rate estimated by MEND. It is difficult to determine whether the conformal symplectic constraint provides any predictive gain beyond contraction-rate calibration alone.

Questions for authors

Q1 Please clarify whether Table 1 uses a probe fitted with the first 60% of the data or a probe fitted using the entire video, and specify the data ranges used for contraction-rate estimation and predictive evaluation. If all model fitting is restricted to the training portion of the data, how do the various methods perform on the independent test set?

Q2 Can the improvements offered by MEND over TLS-DMD and fb-DMD be replicated across a broader range of independent trajectories? It is recommended to compare the amplitude decay and trajectory prediction errors using the same data split for all methods.

Q3 If an existing DMD predictor is calibrated using the contraction rate measured by MEND to achieve the same contraction rate as MEND, how do its predictions compare with those of MEND?

Reviewer_BdEMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MEND, a framework for diagnosing and constraining latent dynamics in world models using frequency, Lyapunov exponents, and contraction rates. It connects representation collapse and prediction drift to encoder rank and predictor spectrum through a semi-conjugacy formulation. Experiments with V-JEPA 2 and physical systems show that a conformal-symplectic predictor can extend the prediction horizon of a real pendulums.

Critical strengths

separation of representation collapse from dynamical instability is mathematically meaningful. use of Lyapunov spectra, symplectic structure, and coordinate invariant contraction provides a principled framework for analyzing long term prediction. connection between theoretical analysis and experiments on physical systems is particularly interesting.

Critical weaknesses

main concern is the gap between the theoretical framework and its practical demonstration. theory largely concerns linear symplectic systems, while the strongest empirical result involves a 2 dimensional predictor fitted to a single pendulum trajectory. This does not establish that the proposed constraints can improve general high dimensional, action conditioned world models. The reported 31 fold improvement also needs careful interpretation. The comparison replaces a large neural predictor with a low-dimensional, physically constrained model using an explicitly selected observable. Also, TLS-DMD and forward-backward DMD already achieve similar longhorizon behavior, while randomly initialized encoders recover the oscillator frequency nearly as accurately as the pretrained encoder. These observations weaken the evidence for a distinctive advantage of proposed over established system-identification methods.

Questions for authors

What improvement does MEND provide over TLSDMD and forward-backward DMD beyond the small difference in estimated damping? Can the proposed constraints be applied directly to a higher dimensional JEPA predictor without constructing a manually selected physical observable? How much of the reported improvement depends specifically on the pretrained representation rather than generic visual features?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.