acceptodds
Under review as a conference paper at ICLR 2027

Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models

Abstract

Robotic systems often exhibit unstable modes, along which small perturbations and disturbances can cause unbounded growth unless corrected through feedback. Controlling such systems from high-dimensional visual observations requires representations that preserve these modes. Joint-embedding predictive architectures (JEPAs) provide a natural framework for learning representations and their dynamics from visual data. However, we demonstrate that next step prediction combined with anti-collapse regularization provides no such guarantee: the training loss can be minimized while controllable unstable modes are collapsed, making stabilization from the learned representation impossible. To circumvent this, we augment predictive world-model training with an action reconstruction objective (i.e., an inverse dynamics loss) that encourages control-aware representations, namely, visual representations that preserve crucial features for control. We prove that exact action reconstruction makes the encoder injective on the finite-horizon reachable subspace, hence, no state direction produced by an action sequence within steps can be discarded. Moreover, as grows, the dominant eigenspace of the finite-horizon controllability Gramian reveals the controllable unstable subspace. Despite our theory considering linear systems, experiments on nonlinear visual control tasks (CartPole, Walker2D, and PointMaze) validate our findings and show the benefits of control-aware representation learning.

Primary area:
learning on time series and dynamical systems
Keywords:
joint-embedding predictive architectures (JEPAs), world models, inverse dynamics, feedback stabilization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NV9jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether JEPA world models preserve the information needed to stabilize unstable systems. The authors show that standard prediction objectives and anti-collapse regularization may fail to preserve unstable modes. To address this issue, they propose Endpoint Inverse Dynamics (EP-IDM), which reconstructs action sequences from the initial and final latent states. The paper provides theoretical guarantees for linear systems and evaluates the method on CartPole, Walker2D, and PointMaze, showing improvements in latent-LQR stabilization and the preservation of walking dynamics.

Critical strengths
  1. A meaningful distinction between predictive accuracy and suitability for feedback control. The paper identifies a specific and consequential failure mode: a representation can remain non-collapsed and support accurate latent prediction while discarding the unstable modes needed for stabilization. The linear counterexample demonstrates that prediction and SIGReg can simultaneously attain their minima without preserving these modes. This is a substantive conceptual contribution because it exposes a limitation of the training objectives themselves, rather than merely documenting an optimization failure or insufficient model capacity.

  2. A principled intervention supported by targeted control diagnostics. EP-IDM provides a simple, control-motivated training objective, and the exact-reconstruction result connects it to the preservation of finite-horizon reachable directions. The CartPole evaluation goes beyond aggregate success rates by examining local dynamics, empirical regions of attraction, and Lyapunov-function changes. Together, these results provide useful evidence that action reconstruction can improve the suitability of learned models for local feedback control, even though the broader interpretation requires further validation.

Critical weaknesses
  1. The experiments do not sufficiently establish the proposed mechanism or its broader applicability. Although three tasks are evaluated, CartPole provides most of the direct evidence for nonlinear stabilization. LQR failure alone does not establish that the encoder has lost unstable information; inaccurate local dynamics could also explain the failure, especially since the SIGReg models perform well with CEM and GBP. The Walker2D results mainly show SAC-driven trajectories within the learned model, which may reflect better fitting of familiar walking trajectories rather than accurate recovery dynamics after disturbances. PointMaze does not test unstable dynamics. Consequently, the evidence for unstable-mode preservation beyond the CartPole setting remains limited.

  2. A substantial gap remains between the theoretical guarantee and practical training. Theorem 1 requires exact action reconstruction, sufficiently rich action excitation, and restrictive dimensionality conditions. In practice, EP-IDM is used as a soft objective without requiring zero reconstruction error. Different action sequences can also produce identical endpoints, making exact recovery impossible in some settings even without representation compression. The paper does not establish when reducing an approximate reconstruction loss reliably preserves unstable information. This weakens the connection between the theoretical result and the practical method.

Questions for authors
  1. What evidence distinguishes unstable information being lost in the encoder from inaccurate local dynamics in the predictor? Can the SIGReg representation support stabilization after refitting the predictor while keeping the encoder fixed?

  2. Could the Walker2D evaluation include a mid-rollout perturbation and compare the predicted recovery with the corresponding MuJoCo recovery under the same SAC policy? This would help distinguish accurate recovery dynamics from simply reproducing a normal gait.

  3. Which experimental settings satisfy the feasibility and action-excitation conditions of Theorem 1? When reconstruction error is nonzero, what evidence connects a lower EP-IDM loss to better preservation of unstable information?

  4. Which physical-state variables are provided as proprioceptive inputs? Do the available vision-only, proprioception-only, and combined-input results establish whether the improvement comes from better visual representations or from the supplied state information?

4 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer NV9j (1/2)

We thank the reviewer for taking the time to review our work and for their constructive feedback. We appreciate the reviewer's perspective on our counterexample as an important step towards highlighting the limitation of the standard JEPA-style training objectives for latent stabilization, as well as the importance of our reachable-subspace guarantee, and the proposed control diagnostics beyond simply success rates.

1. Scope of our mechanism and nonlinear evidence (Weakness 1, Question 1). Lemma 2 proves that encoder information loss can make stabilization impossible even with an exact predictor. In nonlinear CartPole, the LQR failure and the multiple accompanying diagnostics and ablations (supplemental Figs. 9–16 and 19 and Tables 11–14) evaluate the learned encoder-predictor pair. They do not evaluate the literal encoder collapse, as preserving the information needed for control means keeping the directions feedback needs to observe and correct (encoder), and have those directions aligned with the physical ones in the latent space (predictor). We note that performing well with CEM for the CartPole task with models trained with SIGReg does not mean that the learned model reproduces well the evolution of the physical system, especially locally around the unstable equilibrium (that's evaluated by the LQR).

LQR is a “cheaper” method for feedback stabilization for CartPole, as CEM requires repeated sampling and model rollouts. On Walker2D, the SIGReg models also perform poorly with CEM.

Moreover, Figures 5 and 6 and 9–16 evaluate local dynamics and stability, Fig. 19 evaluates rollout error, and Tables 11–14 evaluate state-only predictors, sensory inputs, encoder sensitivity, and fixed-point consistency. We will keep the distinction between the exact linear dynamics failure and the empirical improvement in nonlinear feedback control. We have also provided ablations on a fixed encoder (DINOv2 and iBOT) when adapted with our proposed loss and with SIGReg (Table 4 and supplemental Table 12).

2. Walker2D and PointMaze (Weakness 1, Question 2).

The Walker2D experiment in our submission evaluates whether closed-loop model rollouts preserve the SAC-induced gait. We have included new experiments to evaluate the recovery after a mid-rollout perturbation. We inject a torso-velocity perturbation (δθ˙=±6 rad/s\delta\dot{\theta}=\pm6\,\text{rad/s}) at step t=50t=50 of each episode and roll each model's predictor forward for 80 steps using the ground-truth SAC actions (open-loop), decoding latent states via the learned MLP probe. We report two errors relative to the physical states: predictor MSE, computed by decoding the predicted latents, and re-encoding MSE, computed by encoding and decoding the actual post-perturbation frames. EP-IDM reduces re-encoding MSE from 47.0247.02 to 13.0313.03, indicating that the encoder and probe recover the physical state more accurately from perturbed observations.

ModelPredictor MSE↓Re-encode Ceiling↓
1SP+SIG20.76±7.0620.76 \pm 7.0647.02
1SP+EP-IDM19.56±6.90\mathbf{19.56 \pm 6.90}13.03\mathbf{13.03}

We have also included additional experiments for the Walker2D planning performance:

ModelAvg Velocity (m/s)Displacement (m)Final Height (m)
Ground truth (GT)3.6114.431.21
1SP + SIG0.080.060.98
MSP + SIG0.460.660.85
1SP + EP-IDM2.919.150.92

We deliberately included PointMaze to verify that our proposed EP-IDM objective is still effective on tasks without unstable modes.

AuthorsOpenReview ↗

Point-to-point response to Reviewer NV9j (2/2)

3. Feasibility, excitation, and approximate reconstruction (Weakness 2, Question 3).

When different action sequences produce identical endpoints, our nonzero-loss extension (in the general response) compares reconstruction with the unavoidable physical-endpoint loss. Distinct action sequences share a physical endpoint when their difference lies in ker⁡(CH)\ker(\mathcal C_H). These differences produce no endpoint displacement and cannot be recovered even from the full physical state. Therefore, matching the optimal physical-endpoint loss requires ker⁡(FCH)=ker⁡(CH)\ker(F\mathcal C_H)=\ker(\mathcal C_H), so the encoder introduces no additional ambiguity. This is equivalent to ker⁡(F)∩RH={0}\ker(F)\cap\mathcal R_H=\{0\}.

The linear-Gaussian setting discussed in our response to Reviwer UzQg also provides a quantitative margin on this preservation. We can write the affine action estimate as a^=DFCHa+c\widehat{\mathbf a}=DF\mathcal C_H\mathbf a+c, with DD the terminal-latent decoder block and cc the fixed offset. For Δ=LF−Lx⋆\Delta=\mathcal L_F-\mathcal L_x^\star and δ=HΔ/σa2<1\delta=\sqrt{H\Delta/\sigma_a^2}<1, the reconstruction error bounds ∣∣DFCH−CH†CH∣∣2≤δ||DF\mathcal C_ H-\mathcal C_ H^\dagger\mathcal C_ H||_ 2\leq\delta, yielding

\qquad v\in\mathcal R_ H.$$ Thus, small excess reconstruction loss and bounded decoder gain provide a nonzero preservation margin. In the nonlinear experiments, $H=3$ yields $mH=3$, $18$, and $6$ for CartPole, Walker2D, and PointMaze, respectively. Our nonlinear experiments evaluate EP-IDM as a soft objective, as stated immediately after Theorem 1. Although the theorem assumes zero reconstruction loss, in practice we train with gradient-based methods and do not require the loss to get to zero. The bound also above explains why nonzero loss can still preserve reachable directions in the linear setting. **4. Proprioceptive inputs and modality ablations (Question 4).** The proprioceptive inputs are the four-dimensional CartPole state $(p,\dot p,\theta,\dot\theta)$, the standard $17$-dimensional Walker2D-v4 observation (torso height/orientation, six joint angles, and nine generalized velocities), and the four-dimensional PointMaze position/velocity state (supplemental Table 5). Supplemental Table 13 directly tests what happens when proprioception is removed. With the same frozen iBOT encoder and proprioceptive inputs, LQR success is $0$\% for MSP, $40$\% for MSP + PR-SIG, and $100$\% for MSP + PR-IDM. When proprioception is removed, MSP + PR-IDM drops from $100$\%, $80$\%, and $70$\% success with LQR, CEM, and GBP, respectively, to $0$\% with all three planners, and the same happens with the SIGReg models. The LQR gain therefore persists when the input modalities are held fixed. In the state-only transformer ablation, EP-IDM also increases LQR success from $20$\% to $80$\% (supplemental Table 11).

Thank you for the clarification and additional analyses. These have addressed some of my concerns, although I remain uncertain about the respective roles of encoder information loss and predictor inaccuracies in explaining the nonlinear results. Overall, I am raising my rating and lean toward acceptance based on the theoretical insight and practical benefits.

We thank the reviewer for their time and the positive assessment of our work and for raising their score. We really appreciate that.

Reviewer_8yjQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tried to investigate the preservation of unstable modes in JEPA world models for feedback stabilization. The authors demonstrate that standard prediction and anti-collapse objectives do not necessarily preserve unstable modes by using a linear counterexample. They propose an endpoint inverse dynamics objective (EP-IDM) to encourage control-relevant representations by reconstructing action sequences from initial and final latent states. The theoretical analysis establishes conditions under which exact action reconstruction preserves reachable directions and examines the relationship between the controllability Gramian and unstable subspaces. Experiments on CartPole, Walker2D, and PointMaze demonstrate improvements in learned dynamics and downstream control performance, particularly in latent-LQR stabilization.

Critical strengths

Strengths

  1. The distinction between predictive accuracy and the information required for feedback stabilization is important. The paper brings a useful control-theoretic perspective to JEPA representation learning, and Lemma 2 illustrates a potential limitation of standard prediction and anti-collapse objectives.

  2. The proposed inverse dynamics objective demonstrates meaningful benefits. In particular, the improvement in CartPole latent-LQR stabilization and the better preservation of Walker2D dynamics are encouraging. These results suggest that inverse dynamics provides useful additional supervision for learning control-relevant latent world models.

Critical weaknesses

Weaknesses

  1. The central motivation, proposed objective, and empirical results are not convincingly connected. My main concern is not whether EP-IDM is effective, but whether unstable-mode preservation is indeed the mechanism behind its improvements. I find this connection insufficiently established at three stages.

    (1) Motivation: Is unstable-mode collapse actually the problem? Lemma 2 shows that standard JEPA objectives can discard unstable modes, but this is an existence result based on a constructed linear system. It remains unclear whether such collapse actually occurs in practical world models. In particular, unstable modes can strongly influence system evolution and long-horizon prediction, potentially making them important features to retain. The paper does not sufficiently establish that unstable-mode collapse is responsible for the baseline failures.

    (2) Method: Does EP-IDM specifically encourage unstable-mode preservation? The EP-IDM objective encourages the encoder to retain information useful for reconstructing actions, but it remains unclear why this should specifically favor unstable modes. Theorem 1 establishes preservation of finite-horizon reachable directions under exact reconstruction, while Theorem 2 shows that the dominant eigenspace of the controllability Gramian approaches the unstable subspace as the horizon increases. However, the latter does not establish that minimizing EP-IDM preferentially preserves these dominant directions, particularly when exact reconstruction is infeasible due to limited latent dimensionality. Consequently, although the theoretical results provide useful intuition, the connection between the practical training objective and unstable-mode preservation remains insufficiently explained, especially for nonlinear systems. Thus, while inverse dynamics provides a reasonable source of additional supervision, the paper does not convincingly explain why it should specifically promote unstable-mode preservation rather than more generally improving latent representation and dynamics learning.

    (3) Experiments: Are the improvements attributable to unstable-mode preservation? Although EP-IDM improves control performance, the experiments do not clearly establish the proposed mechanism. In Table 1, SIGReg baselines fail with LQR but achieve high success rates with CEM and GBP. The authors reasonably explain this through LQR's dependence on accurate local linearization. Figure 2 shows that EP-IDM better captures the true unstable directions in the learned predictor, while Figure 5 demonstrates more accurate local dynamics prediction. However, these results do not clearly distinguish whether the baseline encoder has discarded unstable-state information or whether the learned predictor models this information inaccurately. Consequently, it remains unclear whether unstable-mode preservation, rather than improved dynamics identification, explains the gains in stabilization performance.

Overall, the empirical benefits are encouraging, but what information EP-IDM actually preserves, and whether unstable-mode preservation explains its effectiveness, remain unclear.

  1. Limited comparison with alternative representation-learning objectives. The comparisons with SIGReg and other inverse dynamics variants are useful, but it remains unclear whether EP-IDM offers benefits specific to unstable-mode preservation or simply improves latent model learning through additional supervision. Comparisons with other auxiliary objectives, such as reconstruction or task-related losses, would help clarify this distinction. Importantly, such comparisons should evaluate not only control performance but also unstable-mode preservation and local dynamics accuracy, to better identify the mechanism behind the improvements.
Questions for authors

I find the research direction interesting and the empirical results encouraging. EP-IDM appears to be a useful objective for learning better latent world models, and this is a meaningful contribution. However, the paper places unstable-mode preservation at the center of its motivation and theoretical justification, while the experiments do not convincingly establish that this is the mechanism responsible for the observed improvements. I believe the results are worth pursuing, but the paper needs a clearer understanding of what information EP-IDM actually encourages the model to retain and whether its benefits are specifically related to unstable modes, rather than more general improvements in latent dynamics learning.

  1. What evidence establishes that unstable-mode collapse is the actual cause of the baseline failures? Lemma 2 demonstrates that such collapse is possible, but the poor latent-LQR performance could also arise from inaccurate local dynamics. Could the authors directly evaluate whether the baseline encoders discard unstable-state information, particularly in nonlinear CartPole?

  2. Why should minimizing EP-IDM preferentially preserve unstable modes? Theorems 1 and 2 establish different properties of action reconstruction and controllability, but their connection to the actual optimization objective remains unclear. Can the authors clarify this relationship, particularly for nonlinear world models?

  3. Can the authors distinguish unstable-mode preservation from improved dynamics identification? Figure 5 suggests that EP-IDM learns more accurate local dynamics, which could independently explain the improved LQR performance. Could the authors disentangle these effects, for example through direct encoder diagnostics or comparisons with alternative auxiliary learning objectives?

5 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer 8yjQ (1/2)

We thank the reviewer for taking the time to review our work and for their constructive feedback. We appreciate the reviewer pointing out the importance of the latent stabilization question, the control-theoretic perspective of our paper, and our clear improvements in CartPole and Walker2D latent stabilization and locomotion.

1. Why the existence result matters (Motivation, Question 1). Lemma 2 proves that the standard JEPA objective can acheive its global minimum while discarding the unstable modes. Increasing the prediction horizon does not rule out this solution because the kept stable component is predicted exactly at every horizon.

In linearized CartPole, Fig. 2 shows that the unstable directions of the learned dynamics are better aligned with the physical unstable directions under EP-IDM than under SIGReg. EP-IDM also succeeds at stabilization in this comparison, whereas SIGReg fails. In nonlinear CartPole, EP-IDM increases latent-LQR success from 00% to 100100% (Table 1), alongside improvements in local dynamics and stability (Figs. 5–6 and supplemental Figs. 9–16). These experiments evaluate the encoder and predictor jointly, namely, we evaluate whether the learned latent space is good enough to learn a latent dynamics that reproduces well the evolution of the physical open-loop unstable system (see Figs. 5–6 and supplemental Figs. 9–16).

2. What EP-IDM preserves (Method, Question 2). EP-IDM does not preferentially preserve unstable modes. It preserves all action-induced directions, and the unstable directions are among them. Latent prediction can remain accurate after the encoder removes a physical direction, because the prediction target removes it too. EP-IDM instead asks the model to recover the applied actions from their effects on the system. Discarding an action-induced change makes this reconstruction harder, providing supervision that latent prediction alone lacks. This includes unstable directions, where action effects can grow over time, but also other directions needed to distinguish the effects of different actions.

Theorem 1 proves this preservation under its stated linear-system, excitation, and exact-reconstruction assumptions. That is, Theorem 1 proves that ker⁡(F)∩RH={0}\ker(F)\cap\mathcal R_H=\{0\}. Thus, when the unstable subspace is contained in RH\mathcal R_H, the encoder cannot discard any unstable direction. This guarantee comes from Theorem 1. Theorem 2 complements it by characterizing the geometry of the physical action response.

Our experiments support this: Fig. 2 shows better alignment with the physical unstable directions, Figs. 5 and 6 show more accurate local dynamics and improved stability, and Table 1 shows latent-LQR success increasing from 00% to 100100%. Thus, EP-IDM encourages the encoder-predictor pair to preserve action effects needed for feedback control. Our general response above also explains how this preservation can hold with nonzero reconstruction loss.

3. Preservation versus local dynamics identification (Experiments, Questions 1 and 3).

Our linear counterexample characterizes the representation loss (i.e., discarding the unstable modes). The prediction is exact, yet latent stabilization is impossible. Fig. 2 complements this result by showing better alignment with the physical unstable directions and successful stabilization with EP-IDM, whereas models fitted with SIGReg fail to stabilize. This figure evaluates the learned dynamics and control behavior, while Lemma 2 characterizes encoder information loss analytically. In nonlinear CartPole, our diagnostics evaluates the encoder-predictor pair. CEM/GBP success shows that the jointly learned model supports those planners, even when its local dynamics do not support LQR. LQR provides a “cheaper” approach feedback stabilization for CartPole, while CEM requires repeated sampling and model rollouts. On Walker2D, the SIGReg models also perform poorly with CEM. Figures 5 and 6 and 9-16 examine local dynamics and stability. Fig. 19 reports rollout error. Table 14 examines fixed-point residuals. Table 11 demonstrates that an informative input does not replace accurate predictor fitting. Our results establish the EP-IDM benefit for feedback stabilization. We have also provided ablations on a fixed-encoder (DINOv2 and iBOT) in Table 4 and supplemental Table 12.

AuthorsOpenReview ↗

Point-to-point response to Reviewer 8yjQ (2/2)

4. Alternative auxiliary objectives.

In Appendix A.3.4 and Table 8, we compare different inverse-dynamics variants. In Table 1, we include EP-IDM combined with SIGReg, and in Table 4 we also compare frozen pretrained encoders and DINO-WM. Our general response also extends the counterexample from Lemma 2 to VICReg-style regularization and EMA/stop-gradient representation collapse avoidance. We agree that exact physical-state reconstruction would also ensure injectivity. However, we highlight that the premises of JEPA-style world models is to never reconstruct state or pixel space observations. This is the reason we propose EP-IDM that uses actions already available in world-model training and provides the reachable-subspace guarantee. Our main contribution is this control interpretation and its evaluated benefits. We emphasize that the paper does not claim superiority over other auxiliary objective.

Thank you for the detailed response. I appreciate the additional theoretical analysis and clarifications.

My main concern is that the paper's title, motivation, and overall narrative are centered on unstable-mode preservation, but the evidence does not convincingly support this mechanism. The experiments neither establish that the baseline methods fail because their encoders discard unstable information, nor demonstrate that EP-IDM's improvements are attributable to preserving it rather than better dynamics identification. Moreover, the authors themselves clarify that EP-IDM does not preferentially preserve unstable modes. In short, I recognize that this method helps improve latent representations and benefits downstream control tasks, but I remain unconvinced that these improvements can be attributed to unstable-mode preservation, which is central to the paper's motivation and claims.

Furthermore, the proposed approach is not particularly novel in itself, given the extensive literature on inverse dynamics, including multi-step variants, for representation learning. The theoretical analysis is therefore an important part of the contribution, but its guarantees are restricted to linear systems and linear representations under specific assumptions, whereas the experiments involve nonlinear world models with neural encoders and predictors. The connection between these theoretical guarantees and the practical improvements remains unclear.

Overall, while I acknowledge the empirical benefits and the additional theoretical results, I do not think they sufficiently support the central claims of the paper in its current form. I therefore maintain my original score.

Reviewer_UzQgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether JEPA world models retain the information needed to stabilize a physical system. It shows that accurate latent prediction and anti-collapse regularization can still allow an encoder to discard unstable modes. The proposed method adds an endpoint inverse-dynamics head that reconstructs the intervening action sequence from the initial and final latent states, trained jointly with the encoder and forward predictor. For linear systems, the authors prove that exact action reconstruction under sufficient action excitation preserves all directions reachable within the chosen horizon. They also relate the dominant directions of the controllability Gramian to the unstable subspace. The strongest result is improved CartPole stabilization with a controller built from the learned local linear dynamics; gains with nonlinear planners are less pronounced.

Critical strengths

The paper identifies a concrete failure of predictive representation learning: prediction loss and anti-collapse regularization can both be minimized while the encoder discards every unstable mode. This is a useful distinction between a non-collapsed representation and one sufficient for stabilization and gives a clear reason to evaluate world models beyond prediction accuracy.

The paper connects an actionable training objective to a precise control property. Under the stated linear-system and exact-reconstruction assumptions, endpoint action reconstruction preserves the finite-horizon reachable subspace. The CartPole results support its practical relevance: latent-LQR success improves from 0% to 100%, with additional diagnostics showing better local dynamics and a larger region of attraction.

Critical weaknesses
  1. Different action sequences can reach the same endpoint even when the full physical state is available. Increasing the horizon to cover the unstable subspace can therefore make exact reconstruction impossible. Theorem 1 does not explain what the method preserves in this unavoidable nonzero-loss setting.

  2. larger action-induced changes do not necessarily make a direction more valuable for action reconstruction. With fixed initial states and isotropic Gaussian actions, optimal linear reconstruction error depends on the rank of the encoded action map, not its nonzero singular values. The loss can therefore treat stable and unstable directions equally. The link between Theorem 2 and the proposed training objective remains missing.

  3. The encoder can shrink a direction arbitrarily while the inverse decoder compensates with larger weights. Exact reconstruction still holds, but provides no protection against latent errors or numerical precision limits. This is particularly relevant when EP-IDM replaces SIGReg; the role of normalization or weight regularization needs clarification.

  4. LQR is solved in 192 latent dimensions, although valid CartPole states occupy a much smaller space. The predictor may have spurious unstable dynamics outside that space, causing LQR to fail despite an informative encoder. Fitting a separate local model with the encoder frozen, or restricting control to physically valid directions, would help identify the actual cause.

  5. Training includes LQR and SAC trajectories, and physical-state inputs can help predict their actions without using the endpoint. An initial-state-only baseline and separate results on random-action trajectories would clarify whether the improvement comes from preserving action effects or learning the policy that generated the data.

Questions for authors
  1. For the linearized CartPole experiment, does the chosen horizon actually cover all unstable and marginal modes while also satisfying the rank condition for exact action reconstruction? I think it would help to report these ranks explicitly. Otherwise, it is unclear whether Theorem 1 applies to the main stabilization example.

  2. I am not sure how the Gramian result translates into a preference for unstable directions during training. With fixed initial states and isotropic Gaussian actions, the optimal linear inverse error depends on rank, not the size of the nonzero singular values. So, what makes the model prefer unstable directions here -- the prediction loss, regularization, or the training data?

  3. For the sigreg models, how do we know that lqr fails because the encoder discarded unstable information? could the information still be there, but the predictor has learned the wrong local dynamics? Also, since LQR operates in 192 latent dimensions, could the failure come from directions that do not correspond to valid physical states? I think this distinction matters for the main claim.

  4. Another question I had was what stops the encoder from making an unstable direction arbitrarily small while the inverse decoder compensates with larger weights? The action reconstruction could still be exact, right? - but that direction could be very sensitive to latent errors. does the regularization used in practice control this, or does the theorem only guarantee that the information is technically present?

  5. Finally, on the policy-generated trajectories - how much action information comes from the initial representation alone? Also like does the existing results on randomly forced trajectories show that the endpoint provides information beyond predicting the behavior policy? I think this matters for attributing the gain to preservation of action-induced state changes

5 comments
AuthorsOpenReview ↗

Point-to-point response to Reviewer UzQg (1/2)

We thank the reviewer for taking the time to review our work and for their constructive feedback. We appreciate the reviewer pointing out the importance of our results in highlighting the distinction between a non-collapsed representation and a meaningful representation that is later used for feedback stabilization. We also appreciate their recognition of the importance of our reachable-subspace result and our numerical validation.

1. Exact reconstruction, and approximate reconstruction (Weakness 1, Question 1).

Theorem 1 states the feasibility condition rank⁡(FCH)=mH\operatorname{rank}(F\mathcal C_H)=mH for exact action reconstruction. When physical endpoints cannot uniquely determine the action sequence, Eqs. (1)-(2) (in our general response) characterize preservation relative to the unavoidable physical-endpoint reconstruction error.

Distinct action sequences share a physical endpoint when their difference lies in ker⁡(CH)\ker(\mathcal C_H). These differences produce no endpoint displacement and cannot be recovered even from the full physical state. Matching the optimal physical-endpoint loss requires ker⁡(FCH)=ker⁡(CH)\ker(F\mathcal C_H)=\ker(\mathcal C_H), so the encoder introduces no additional ambiguity. This is equivalent to ker⁡(F)∩RH={0}\ker(F)\cap\mathcal R_H=\{0\}. Theorem 1 treats exact recovery under its feasibility condition. The extension in our general response allows physically ambiguous endpoints and nonzero reconstruction loss.

For stabilization, however, preserving the entire reachable space is stronger than necessary, this is why we additionally note that ker⁡(F)∩Φ≥1={0}\ker(F) \cap \Phi_{\geq 1} =\{0\}, i.e., the representation must keep the unstable and marginal modes. For linearized CartPole, these span a three-dimensional subspace, so the three-dimensional latent representation can preserve them while discarding the strictly stable direction. The relevant check is then rank⁡(Fφ≥1)=3\operatorname{rank}(F\varphi_{\geq1})=3, where φ≥1\varphi_{\geq1} spans this subspace, and the unstable mode alignment which is given in Fig. 2. We will make this clearer in our paper.

For the nonlinear CartPole, EP-IDM is used as a soft training objective with H=3H=3. Table 1 shows that this objective increases latent-LQR success from 0%0\% to 100%100\%, demonstrating its benefit beyond the exact-reconstruction of Theorem 1.

2. Rank versus singular values (Weakness 2, Question 2).

Our preservation mechanism is described in Eq. (2):

\left[ \operatorname{rank}(\mathcal C_H) -\operatorname{rank}(F\mathcal C_H) \right].$$ Therefore, discarding any reachable direction would introduce an additional reconstruction error of at least $\sigma_a^2/H$. This applies to unstable directions contained in $\mathcal R_H$, regardless of the magnitudes of the nonzero singular values. Hence, a preference among the directions to keep is unnecessary for Theorem 1's preservation guarantee, as it will preserve the reachable directions which include the crucial unstable ones for effective feedback stabilization. Theorem 2 characterizes which directions in the physical state space exhibit the largest variation at the endpoint when we vary the actions while holding the initial state fixed. For a fixed initial state and isotropic actions, we have $\operatorname{Cov}(x_H\mid x_0)=\sigma_a^2\Pi_H$. Therefore, under the theorem's assumptions, the dominant eigenspace of this covariance approaches the unstable subspace as $H$ grows. We will state this distinctions more clearly in the text, i.e., Theorem 1 establishes the preservation of the unstable directions, while Theorem 2 identifies the dominant physical action responses which are precisely those unstable directions.
AuthorsOpenReview ↗

Point-to-point response to Reviewer UzQg (2/2)

3. Scaling and robustness (Weakness 3, Question 4).

The scaling can be quantified directly. In the fixed-initial-state linear-Gaussian setting of our general response, we can write the affine action estimate as a^=DFCHa+c\widehat{\mathbf a}=DF\mathcal C_H\mathbf a+c, where DD is the decoder block multiplying the terminal latent and cc includes the decoder bias and the contributions of the fixed initial state. We can then define Prow:=CH†CHP_{\mathrm{row}}:=\mathcal C_H^\dagger\mathcal C_H. For an action reconstruction loss LF\mathcal L_F, let Δ=LF−Lx⋆\Delta=\mathcal L_F-\mathcal L_x^\star. The orthogonal decomposition of the reconstruction error provides

HΔ=σa2∣∣DFCH−Prow∣∣F2+∣∣c∣∣22.H\Delta =\sigma_a^2||DF\mathcal C_H-P_{\mathrm{row}}||_F^2 +||c||_2^2.

Then, by setting δ=HΔ/σa2<1\delta=\sqrt{H\Delta/\sigma_a^2}<1, we obtain

∣∣DFCH−Prow∣∣2≤δ.||DF\mathcal C_H-P_{\mathrm{row}}||_2\leq\delta.

For any v∈RHv\in\mathcal R_H, we can then apply this bound to u=CH†vu=\mathcal C_H^\dagger v, for which Prowu=uP_{\mathrm{row}}u=u and CHu=v\mathcal C_Hu=v. By the triangle inequality, we can write

∣∣DFv∣∣2=∣∣DFCHu∣∣2\displaystyle ||DFv||_2 = ||DF\mathcal C_Hu||_2

≥∣∣Prowu∣∣2−∣∣(DFCH−Prow)u∣∣2\displaystyle \geq ||P_ {\mathrm{row}}u||_ 2 - ||(DF\mathcal C_ H-P_ {\mathrm{row}})u||_ 2

≥(1−δ)∣∣u∣∣2.\displaystyle \geq (1-\delta)||u||_2.

Then, by combining this with ∣∣DFv∣∣2≤∣∣D∣∣2∣∣Fv∣∣2||DFv||_2\leq||D||_2||Fv||_2 and ∣∣v∣∣2=∣∣CHu∣∣2≤∣∣CH∣∣2∣∣u∣∣2||v||_2=||\mathcal C_Hu||_2 \leq||\mathcal C_H||_2||u||_2 yields

\tag{4}$$ **4. Encoder versus predictor and off-manifold dynamics (Weakness 4, Question 3).** We emphasize that Lemma 2 characterizes the representation loss (i.e., discarding the unstable modes) with an *exact* predictor, yet discarding the unstable modes makes latent stabilization impossible. For nonlinear CartPole, we provide several ablations and diagnostics: the vector-field, prediction-error, region-of-attraction, and Lyapunov certificate diagnostics establish the benefit of EP-IDM for the learned encoder-predictor pair. Table 11 also shows that predictor architecture matters, namely, with state-only inputs, EP-IDM achieves $80$\% LQR success with the transformer predictor but completely fails with the MLP predictor. Hence, our theory characterizes the representation degenerate solution and guarantees for preserving the reachable directions in the latent space, and the nonlinear experiments evaluate whether the jointly learned model supports stabilizing feedback. **5. Behavior-policy for collecting the data (Weakness 5, Question 5).** The data is not limited to expert trajectories, i.e., $75$\% of CartPole and Walker2D trajectories use passive or uniformly random inputs. These reduce the dependence on behavior-policy prediction. We note that, for independently sampled zero-mean actions, the optimal initial-state-only predictor has loss $m\sigma_a^2$, so improving on this baseline requires information beyond the initial state. In Theorem 1, we explicitly assume conditional action excitation for this reason.

Thanks for the detailed response. The clarification on approximate reconstruction is helpful, and I think the distinction between Theorem 1 and Theorem 2 is much clearer now.

I still have a couple of concerns:

  1. On the CartPole example, I am not seeing how the theoretical guarantee applies at H=3. From the matrices in the appendix, the unstable direction doesn't seem to lie in the reachable subspace at this horizon. So even if rank(FC_3) = 3, it doesn't necessarily mean the unstable mode is preserved. Could the authors verify this explicitly?

  2. On the encoder vs predictor issue, I agree EP-IDM improves the learned model, but how do we know SIGReg actually loses unstable information? Could it be that the information is still there, but the predictor learns incorrect dynamics? I think testing both frozen encoders with the same fitted local dynamics model would help separate these two effects.

  3. Also, on the scaling argument, the new bound still depends on the decoder norm. So unless that is controlled, couldn't the encoder still shrink a direction arbitrarily while the decoder compensates?

Overall, the rebuttal addresses some of my theoretical concerns, but I am still not fully convinced about the main representation-preservation claim, especially how it connects to the experimental results. For now, I'll keep my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the Reviewers (1/2)

We thank the reviewers for their constructive feedback and for recognizing the importance of our results in highlighting the difference between achieving a good latent predictive accuracy and actually learning a meaningful representation that can be used for latent feedback stabilization. The reviewers also recognize our rigorous reachable-subspace guarantee, and the benefits of our proposed inverse dynamics loss on CartPole and Walker2D as major strengths of the paper. We first address the comments that are shared between the reviewers.

On the significance of our counterexample in Lemma 2.

In Lemma 2, we establish the fundamental limitation of the standard recipe for JEPA-style world models (i.e., latent prediction loss combined with anti-collapse representation regularization). We demonstrated that the encoder can discard every unstable mode of the physical system while achieving both zero prediction error and the minimum anti-collapse regularization. This is already sufficient to disprove any stabilization guarantee for latent dynamics learned with these objectives, even with perfect optimization and unlimited data. Our counterexample construction is deliberately simple so that the failure cannot be attributed to nonlinear optimization or an inaccurate predictor. In addition, a larger prediction horizon HH does not resolve it, namely, the easy-to-predict stable component kept in the latent space admits exact prediction at every horizon. We also emphasize that the unstable modes affect the physical system trajectory but will contribute with no latent prediction error once the encoder has removed them. This is exactly the difficulty for effective latent feedback control, and it motivates proposing additional control-aware supervision for training the encoders and latent predictors in visual world models for open-loop unstable dynamics.

On approximate action reconstruction. Theorem 1 states an exact-reconstruction guarantee and its feasibility condition. A useful extension is to compare the action reconstruction loss with the error that is already unavoidable from the physical endpoints. More precisely, we can fix x0x_0 and suppose that the actions are given by a∼N(0,σa2Iq)\mathbf a\sim\mathcal N(0,\sigma_a^2 I_q) with q=mHq=mH. We also define r=rank⁡(CH)r=\operatorname{rank}(\mathcal C_H). For an affine endpoint decoder, let LF⋆\mathcal L_F^\star denote the minimum of the endpoint action-reconstruction loss LEP-IDM\mathcal L_ {\mathrm{EP}\text{-}\mathrm{IDM}}, with the encoder FF fixed. Similarly, let Lx⋆\mathcal L_x^\star denote the minimum of the same action-reconstruction loss when the decoder receives the physical endpoints (x0,xH)(x_0,x_H) instead of their latent representations (z0,zH)(z_0,z_H). Both losses are evaluated for the fixed initial state x0x_0 and the action distribution as above. The optimal losses are given by

=\frac{\sigma_a^2}{H} \bigl(q-\operatorname{rank}(F\mathcal C_H)\bigr), \qquad \mathcal L_x^\star =\frac{\sigma_a^2}{H}(q-r). \tag{1}$$ Hence, we have $$\mathcal L_F^\star-\mathcal L_x^\star =\frac{\sigma_a^2}{H} \bigl(r-\operatorname{rank}(F\mathcal C_H)\bigr). \tag{2}$$ If an action decoder achieves an endpoint action-reconstruction loss $\mathcal L_ {\mathrm{EP}\text{-}\mathrm{IDM}}$ such that $\mathcal L_ {\mathrm{EP}\text{-}\mathrm{IDM}}-\mathcal L_x^\star<\sigma_a^2/H$, then we have $\operatorname{rank}(F\mathcal C_H)=r$, which implies $\ker(F)\cap\mathcal R_H=\{0\}$. Therefore, preserving the action-induced directions (and unstable directions contained in $\mathcal R_H$) in the latent space can hold even with nonzero reconstruction loss, including when several action sequences share the same endpoint. **On the significance of the Theorem 2.** We emphasize that Theorem 1 characterizes the preservation of reachable directions in the latent space. On the other hand, Theorem 2 characterizes the geometry of the physical action response, namely, under its mild assumptions, the dominant Gramian eigenspace approaches the unstable subspace of the physical system. In particular, Equation (1) makes the preservation mechanism very precise: discarding a reachable direction would strictly increase the optimal reconstruction loss above the physical-endpoint loss. This includes every unstable direction in $\mathcal R_H$, regardless of whether stable directions are also kept. Theorem 2 complements this guarantee by showing that the dominant action-induced directions increasingly align with the unstable directions as the horizon grows. We will clarify this interpretation in the paper by replacing “most identifiable” with “largest action-induced state directions”.
AuthorsOpenReview ↗

General comment to the Reviewers (2/2)

On the experimental validation and ablations.

Our experimental validation goes beyond the theoretical guarantees by evaluating:

  • Learned unstable directions: the linearized CartPole experiment examines alignment with the physical unstable directions (Fig. 2).
  • Nonlinear feedback stabilization: EP-IDM increases CartPole latent-LQR success from 00% to 100100% (Table 1).
  • Local dynamics and stability: vector fields, prediction errors, regions of attraction, and Lyapunov decrease assess whether the learned models support stabilizing feedback (Figs. 5 and 6 and 9–16).
  • Planning beyond CartPole: Walker2D evaluates locomotion and exposes failures of SIGReg models even with CEM, PointMaze evaluation whether EP-IDM remains useful for navigation without an unstable-equilibrium requirement.
  • Objective and action-context ablations: inverse-dynamics variants and different action contexts examine which supervision supports effective control (Tables 8–10).
  • Inputs and architectures: state-only and vision-only configurations, predictor architectures, and frozen DINOv2/iBOT encoders assess how the benefits depend on sensory information and model design (Tables 4 and 11–12).
  • Additional model diagnostics: rollout error, encoder sensitivity, and fixed-point residuals examine prediction accuracy and local consistency (Fig. 19 and Tables 13–14).

These experiments evaluate the practical control benefit of EP-IDM beyond the exact linear setting, using both control performance and dynamics diagnostics. These results are reported in supplemental Tables 8–14 and Figs. 9–16 and 19, alongside Table 4.

On other anti-collapse mechanisms. SIGReg is the regularizer analyzed in the paper, within the standard JEPA recipe of prediction plus collapse avoidance. However, we emphasize that our counterexample in Lemma 2 extends to other anti-collapse mechanisms:

  • VICReg (variance/covariance regularization): For instance, let us choose the stable coordinates in Lemma 2 to have covariance s2Is^2I, with ss above a given variance threshold. The variance and covariance regularization components can then minimized. In addition, latent prediction would still exact while all unstable modes are absent in the designed latent space.
  • EMA/stop-gradient: Let us take identical online (parameter θ\theta) and target (parameter θˉ\bar \theta) encoders Eθ(x)=Eθˉ(x)=FsxsE_ \theta(x)=E_ {\bar\theta}(x)=F_sx_s, as in our counterexample, and the exact predictor P(z,a)=FsAsFs−1z+FsBsaP(z,a)=F_sA_sF_s^{-1}z+F_sB_sa. Then P(Eθ(xt),at)=sg⁡(Eθˉ(xt+1))P(E_ \theta(x_t),a_t)=\operatorname{sg}(E_ {\bar\theta}(x_ {t+1})), so the prediction residual and its gradients are zero. The online parameters remain unchanged, and the EMA update satisfies θˉ+=τθˉ+(1−τ)θ=θˉ\bar\theta^+=\tau\bar\theta+(1-\tau)\theta=\bar\theta, with τ∈[0,1)\tau \in [0,1). Hence, the predictive objective admits a non-collapsed, zero-error fixed point that discards every unstable mode of the physical system. EMA and stop-gradient therefore do not rule out the degenerate solutions from our counterexample.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.