acceptodds
Under review as a conference paper at ICLR 2027

World Modeling Through Spectral Alignment

Abstract

What information should a latent world model preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a teacher similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World models, JEPA, spectral manifold learning, robotics, planning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_g2iLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Building on existing spectral representation learning and kernel alignment frameworks, this paper proposes SpecWM, a joint training approach for visual latent world models that combines spectral alignment with action-conditioned dynamics prediction. It investigates how different teacher kernels shape learned representations through theoretical analysis and linear probing, and evaluates their impact on downstream planning. Across three visual manipulation environments, OGBench Cube, OGBench Scene, and CALVIN, the temporal and quasimetric variants improve CEM planning success over LeWM by approximately 21 percentage points on average.

Critical strengths
  1. The proposed joint objective, combining spectral alignment with action-conditioned latent prediction, achieves consistent improvements in CEM planning success over LeWM across three visual manipulation environments, with an average gain of approximately 21 percentage points using temporal and quasimetric kernels.
  2. The paper empirically investigates how different teacher kernels influence the information preserved in learned representations through linear probing. The results show that kernel choice affects which physical quantities are linearly recoverable, and that the self-supervised temporal kernel can preserve substantial physical state information without requiring ground-truth state supervision.
Critical weaknesses
  1. The planning experiments are limited to three visual manipulation environments, without systematically evaluating generalization to unseen scenes, task compositions, or environment configurations. It remains unclear whether the learned representations can generalize beyond the training environments, particularly given the substantial amount of environment-specific training data required.
  2. The connections between spectral decomposition, contrastive representation learning, and successor measures have been extensively studied in prior work, as partly acknowledged by the authors. Closely related matrix-factorization objectives and multi-step transition operators have also been established in previous studies [1, 2]. The main technical novelty of this paper therefore appears to lie in combining spectral alignment with action-conditioned latent dynamics prediction. The related work section could further clarify the distinction from these earlier approaches.

References:

[1] Wu et al. (2018). The Laplacian in RL: Learning Representations with Efficient Approximations. https://arxiv.org/abs/1810.04586

[2] Tan et al. (2024). Contrastive Learning Is Spectral Clustering on Similarity Graph. https://arxiv.org/abs/2303.15103

Questions for authors
  1. How well do the teacher kernels reflect the state relationships needed for goal-directed planning? While the paper compares temporal and quasimetric kernels, it remains unclear how well the learned similarities reflect goal-conditioned reachability or planning costs. In particular, temporal distances can be affected by pauses, detours, revisits, and varying execution speeds, which do not necessarily change the underlying reachability structure. Could the authors investigate how such discrepancies affect the learned representations and planning performance?
  2. How does the proposed method perform on longer-horizon planning tasks? The planning experiments in Appendix C.3 use relatively short prediction horizons (25 environment steps for OGBench and 50 for CALVIN), with frequent replanning. For longer horizons, errors in recursive latent dynamics prediction may compound, while inaccuracies or biases in the teacher-defined similarities may lead to incorrect rankings of candidate action sequences. Could the authors evaluate planning performance over longer horizons and investigate how these two sources of error affect planning reliability?
3 comments

Thank you for your thoughtful review. We are glad you found the analysis of what each teacher kernel preserves informative and the planning results compelling. Below are responses to each weakness.

W1: The planning experiments are limited to three visual manipulation environments, without systematically evaluating generalization to unseen scenes, task compositions, or environment configurations.

We thank the reviewers for this point. We’ve updated our CALVIN evaluation to train on environments A, B, and C, and evaluate on environment D (Appendix B.1). We find that all methods see a drop in performance on environment D, but SpecWM still outperforms LeWM, and thus display better generalization.

W2: The main technical novelty of this paper therefore appears to lie in combining spectral alignment with action-conditioned latent dynamics prediction. The related work section could further clarify the distinction from these earlier approaches.

We agree - while spectral methods have explored connections to RL, their instantiation in latent world model remains, to the best of our knowledge, novel. As suggested, we have further expanded our related works to not just mention related studies, but also how this work positions with respect to them.

Q1: How well do the teacher kernels reflect the state relationships needed for goal-directed planning?

Thank you for suggesting this analysis, which we are happy to provide. The answer to your question depends on the kernel used: while the temporal kernel should capture temporal distances under the behavioral policy, the quasimetric kernel should relate to optimal temporal distances. While both should correlate with reachability, we expect correlation to be stronger for the latter. We perform the evaluation you suggested in Appendix B.3, and find that the temporal-kernel similarity correlates with distance under the behaviour policy to a Spearman of 0.81 and with the true reachability to a 0.70. Hence, the model learns the behavioural somewhat more strongly than the optimal one, as predicted by Theorem 4. As temporal distance and reachability are in practice often correlated, learned similarities remain nevertheless informative for reachability.

Q2: How does the proposed method perform on longer-horizon planning tasks?

Thank you for the suggestion. We have updated the paper with CEM performance at longer horizons and a plot of prediction and ranking error over horizons in Appendix B.2. We observe that for SpecWM, CEM performance deteriorates gradually, likely stemming from weaker ranking of progress than prediction as compared to LeWM.


Thank you for suggesting these additional analyses and ablation, which we believe have further improved our submission. Please, let us know if any questions remain: we are happy to address them through the rest of the rebuttal phase.

Thank you for your review and your support. As the rebuttal phase draws to a close, please let us know if any question remains; we'd be happy to answer.

Reviewer_g2iLOpenReview ↗

Rebuttal Response

Thank you to the authors for the thoughtful responses and additional experiments. W1 and the related work concern in W2 are largely addressed, although my assessment of limited technical novelty remains unchanged. Q1 is partially addressed, but correlations with reachability weaken at longer distances and across episodes. Q2 is partially addressed, but long-horizon performance remains limited, and the contributions of teacher-related and dynamics errors remain unclear. Therefore, I maintain my weak accept recommendation.

Reviewer_uzmDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which state information latent world models should retain for control and planning, and proposes a spectral-alignment-based world model, SpecWM, which explicitly defines relationships between observations via a teacher similarity kernel and trains an encoder so that the Gram matrix of its latent representations aligns with the target kernel matrix. The framework supports different teacher kernels, including physical states, proprioception, depth information, and a self-supervised temporal kernel that relies only on the temporal order of observations in trajectories. Theoretically, the paper analyzes the global optimum of the spectral alignment objective, the linear recoverability of target variables, and invariance to task-irrelevant information, and establishes a connection between the temporal kernel and the Behavioral Successor Measure. Empirically, the authors evaluate SpecWM on three robot visual manipulation environments, OGBench Cube, OGBench Scene, and CALVIN, and show that it clearly outperforms LeWorldModel in CEM planning success rates, while the choice of teacher kernel significantly affects the extent to which latent representations preserve physical state variables.

Critical strengths
  1. This paper focuses on the fundamental question of what information latent world models should retain, and points out that existing JEPA methods mainly rely on prediction objectives and regularization mechanisms that prevent representation collapse, lacking explicit control over how task-relevant information is retained. SpecWM incorporates different types of supervision signals into a unified framework through spectral alignment between teacher kernels and the Gram matrix of student embeddings; in particular, the temporal kernel constructs supervision only from the temporal order of observations in trajectories, without requiring complete physical state annotations, and thus has good practical potential.

  2. The paper analyzes the properties of the objective from the perspective of spectral representation learning, and provides mathematical results on spectral recovery, linear recoverability, invariance to distractors, and the relationship between the temporal kernel and the Behavioral Successor Measure, with corresponding proofs in the appendix. Empirically, SpecWM achieves clear planning performance improvements in all three robot manipulation environments; for example, the temporal-kernel SpecWM improves the planning success rate from 49% to 77% on OGBench Scene and from 29% to 51% on CALVIN. The authors also provide supplementary ablations on training data, planning distance functions, normalization schemes, and prediction gradients.

Critical weaknesses
  1. SpecWM and LeWorldModel differ not only in their representation objectives (Spectral Alignment vs. SIGReg) but also in their dynamics prediction losses (Cosine vs. MSE), as shown in Appendix C.4 and Table 5. Consequently, the reported planning gains cannot be uniquely attributed to spectral alignment. While Appendices B.4–B.5 examine other implementation choices, they do not control for this key confounder. Moreover, Appendix B.2 shows that prediction-loss gradient propagation affects planning performance. A controlled cross-ablation of the two representation objectives under both prediction losses is necessary to establish the independent contribution of spectral alignment.

  2. Theorem 4 relates the expected temporal-kernel target to the Behavioral Successor Measure under simplified assumptions that differ from the actual training objective, including the removal of centering and spectral adjustments. More fundamentally, the theorem assumes unit-norm embeddings, implying s(x,x)=1s(x,x)=1, whereas its expected target satisfies RT(x,x)→0R_T(x,x)\to 0 as T→∞T\to\infty. Thus, the theoretical target cannot be exactly realized by the assumed encoder in this limit. While this does not invalidate Theorem 4, it leaves unclear whether the learned representations preserve the claimed successor-measure structure. The authors should provide either an approximation analysis under the actual representation constraints or empirical evidence linking learned similarities to successor measures and state reachability.

Questions for authors

Q1. SpecWM and LeWorldModel simultaneously change the representation regularization objective and the dynamics prediction loss. Can the authors provide cross-ablations of SIGReg/Spectral Alignment with MSE/Cosine prediction loss to demonstrate that the planning performance improvement mainly comes from Spectral Alignment rather than from the change in prediction objective?

Q2. Theorem 4 mainly characterizes the properties of the temporal-kernel teacher objective. Considering that a unit-norm encoder cannot exactly match the diagonal entries of this target in the long-trajectory limit, can the authors explain the relationship between the actually learned representations and the Behavioral Successor Measure, and provide corresponding theoretical approximation analysis or empirical verification?

2 comments

Thank you for the thorough review, and in particular for the close reading of Theorem 4 and the appendices. We have addressed both points and have made additions to the paper. We respond to each in turn below.

Q1/W1: Can the authors provide cross-ablations of SIGReg/Spectral Alignment with MSE/Cosine prediction loss to demonstrate that the planning performance improvement mainly comes from Spectral Alignment rather than from the change in prediction objective?

As suggested, we performed an ablation for MSE vs. cosine distance in the prediction loss (Appendix B.7). We find that the difference induced by this change is not statistically significant, suggesting that the performance improvement can be traced back to spectral alignment.

[...] can the authors explain the relationship between the actually learned representations and the Behavioral Successor Measure, and provide corresponding theoretical approximation analysis or empirical verification?

Theorem 4 provides intuition in an idealized, infinite-horizon regime. As the reviewer suggests, the practical algorithm uses unit encoder, which would not be compatible with this regime. We can close this discrepancy in two ways: by pointing out that the practical algorithm operates in a different regime (and extending the analysis accordingly), and by measuring that, empirically, similarities track the behavioral successor measure.

First, we note that the practical setting operates with finite TT (and scales σ\sigma with TT). Thus, it does not approach the regime in which RT(s,s)→0R_T(s,s) \to 0. We can extend the analysis in Theorem 4 to describe this regime, and we have done so in Appendix D.5. In short, truncating the trajectories results in recovering weighted, truncated visitation probabilities.

Second, we empirically verify that latent similarities relate to the behavioral successor measure, as demonstrated by new results in Appendix B.3. SpecWM follows distance under the behaviour policy most closely, consistent with Theorem 4, and orders cross-episode states by reachability better than LeWM, although the temporal kernel never compares frames across episodes; cross-episode correlations vanish beyond 100 steps.

—

Thank you for the constructive and precise feedback. We hope the new ablation and the extended analysis of Theorem 4 address your concerns. Please let us know if anything remains unclear, as we are happy to continue the discussion throughout the rebuttal period.

Thank you for your review and your support. As the rebuttal phase draws to a close, please let us know if any question remains; we'd be happy to answer.

Reviewer_pqM8OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SpecWM. A teacher similarity kernel specifies how observations should relate to one another, and a JEPA-style world model trains its encoder to reproduce those relationships in embedding space while learning an action-conditioned predictor. The teacher can draw on extra information, such as physical state, or use only temporal separation within a trajectory. The theory covers spectral optima and linear recoverability for fully specified kernels, nuisance invariance under coverage and zero-loss assumptions, and a connection between an idealized temporal kernel and the behavior policy's successor measure. In CEM planning, the temporal-kernel model improves success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN.

Critical strengths
  1. The central idea is easy to grasp: specify the similarity structure a representation should preserve, rather than leave it entirely to the prediction loss to emerge on its own. The temporal kernel is especially appealing because it needs trajectory order, not simulator-state labels.

  2. I also appreciate the amount of empirical work here. Planning improves noticeably in three manipulation environments, with 100 fixed tasks per environment, three training seeds, and results from multiple checkpoints. The authors examine linear probes and several teacher kernels rather than focusing only on success rates. They also ablate failure-rollout data, planning distance, whether to stop gradients through the target, and projection-head normalization. Training and planning details are fairly complete, and an anonymous code link is provided. At a minimum, the theory helps explain why changing the kernel can change what information the representation retains.

Critical weaknesses

1.There is still some gaps between the theory and the method that is actually trained. Theorem 4, which is the result most closely tied to control, studies an uncentered, untempered, untruncated population objective for a stationary finite-state Markov chain. The implemented method instead uses centered kernels within trajectories, a neural encoder, and a predictor. I therefore read the successor-measure connection as an interpretation, not as a guarantee that the trained planner approximates goal reachability. The derived target is also symmetrized and depends on the behavior policy, whereas planning decisions are directional and action-dependent.

  1. The assumptions behind Theorem 3 do not quite match the main experiments either. The theorem requires target-diagonal rescaling and zero expected spectral loss; the trajectory-local temporal and quasimetric variants used in practice explicitly omit that rescaling. If this theorem is meant to support the claim that the main method ignores nuisance information, the paper needs an argument for the implemented objective, or a direct test of nuisance sensitivity.

  2. LeWorldModel is the most thoroughly examined comparator. Still, I would like to know how much of the improvement comes specifically from spectral alignment, as opposed to a more general temporal-similarity auxiliary objective or extra computation. On CALVIN in particular, SpecWM processes 744 images per update versus 552 for LeWorldModel. The existing ablations are useful, but this budget difference deserves a direct matched comparison. A simple temporal-pair or contrastive objective, run with the same architecture and budget, would help pin down the contribution.

Questions for authors
  1. On held-out state pairs, how strongly does learned temporal-kernel similarity correlate with actual directional, action-conditioned reachability? What happens when the behavior policy changes?
  2. Can the authors establish nuisance invariance for the trajectory-local objective used in the main experiments, or test its sensitivity to irrelevant noise directly?
  3. On CALVIN, do the planning gains persist if LeWorldModel receives a comparable number of images, or if SpecWM is limited to the same image and wall-clock budget? How do training time and memory use compare?
  4. Under the same architecture and compute budget, how does SpecWM compare with a simpler temporal contrastive or pairwise-distance auxiliary loss?
3 comments

Thank you for your detailed review and encouraging evaluation; we are glad to hear that the central idea is easy to grasp and that the empirical work is appreciated. We will address each weakness and answer each question.

W1: There is still some gaps between the theory and the method that is actually trained. Theorem 4 studies an uncentered, untempered, untruncated population objective for a stationary finite-state Markov chain.

This is correct: Theorem 4 relies on simplifying assumptions, some of which are not easily avoidable: for instance, studying the training dynamics of deep neural networks is generally prohibitive, which restricts analyses to population objectives. Similarly, stationarity is required for studying asymptotic behavior. The absence of tempering is instead consistent with the practical method, which does not implement it for underspecified kernels (e.g. temporal). Other assumptions can in principle be lifted at the cost of a more cumbersome analysis and a less intuitive objective. For instance, under finite trajectories the objective can be related to weighted, truncated visitation probabilities, while target centering the target produces a constant offset. Student centering is harder to study. We remarked this in Section D.5. As the analysis is presented as an interpretation rather than a guarantee, we would prefer keeping these assumptions; we have rephrased Section 5.4 to make this clearer.

Q1: On held-out state pairs, how strongly does learned temporal-kernel similarity correlate with actual directional, action-conditioned reachability? What happens when the behavior policy changes?

Thank you for suggesting this analysis, which we are happy to provide. The answer to your question depends on the kernel used: while the temporal kernel should capture temporal distances under the behavioral policy, the quasimetric kernel should relate to optimal temporal distances. While both should correlate with reachability, we expect correlation to be stronger for the latter. We perform the evaluation you suggested in Appendix B.3, and find that the temporal-kernel similarity correlates with distance under the behaviour policy to a Spearman of 0.81 and with the true reachability to a 0.70. Hence, the model recovers behavioral distances somewhat more strongly than the optimal one, as suggested by Theorem 4. We would thus expect the similarities to reflect changes in the behaviour policy.

Q2/W2: Can the authors establish nuisance invariance for the trajectory-local objective used in the main experiments, or test its sensitivity to irrelevant noise directly?

The nuisance invariance established in the paper applies to the trajectory local objective, it only requires that the pairwise distances within the trajectory are fully specified. As requested, we have included an experiment of nuisance invariance in a noisy-TV scenario in Figure 5. The encoder becomes almost entirely noise invariant in the first few thousand steps, with or without rescaling.

Q3/W3: On CALVIN, do the planning gains persist if LeWorldModel receives a comparable number of images, or if SpecWM is limited to the same image and wall-clock budget?

As suggested, we have rerun LeWorldModel with a matched number of images. We find that the planning benefits persist under matched image and step budgets, and we have updated Figure 3 accordingly.

Q3/W3: How do training time and memory use compare?

We are happy to report time and memory costs of our approach in Appendix C.5. SpecWM’s overhead is negligible (~1%), both in terms of training time and memory.

Q3/W3: Under the same architecture and compute budget, how does SpecWM compare with a simpler temporal contrastive or pairwise-distance auxiliary loss?

Thank you for the suggestion. We added two baselines. Both keep SpecWM’s architecture and change only the representation loss. The first is temporal InfoNCE, trained on the same clips and with the same number of images per update as SpecWM. For each frame, the positive is a later frame of the same clip, sampled with the decay of SpecWM’s temporal kernel, and the negatives are all other frames in the batch, including frames of the same trajectory. Frames are scored by the dot product of their embeddings. The second is contrastive RL (Eysenbach et al., 2022), following the OGBench reference implementation. It samples state and goal pairs independently across episodes, uses goals from other episodes as negatives, and scores pairs with a learned critic on separate state and goal heads. It sees more images per update than SpecWM. Please let us know if these baselines differ from what you had in mind, and we will extend the comparison.


Thank you for providing a precise and informative review. We hope we were able to answer your questions in similar detail. Please, do not hesitate to ask for any additional clarification; we remain available for further discussion through the rest of the rebuttal.

Thank you for your review and your support. As the rebuttal phase draws to a close, please let us know if any question remains; we'd be happy to answer.

Reviewer_pqM8OpenReview ↗

Official Comment by Reviewer pqM8

Than you for your detailed response. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.