acceptodds
Under review as a conference paper at ICLR 2027

Marginal normality does not constrain time: a prescribed Gaussian path law for JEPA world models

Abstract

SIGReg, the anti-collapse regulariser of the LeWM world model, constrains the latent marginal at each time step and says nothing about how consecutive steps relate: a latent that is constant along every trajectory attains the global minimum of both training terms, and the identifiability theory for this objective assumes exactly the cross-time structure it never enforces (Klindt et al., 2026). We close the gap with **T-SIGReg** (temporal SIGReg), which prescribes the joint law of a trajectory window, , with the profile measured from the observation stream before training, and enforces it with the characteristic-function test SIGReg already uses, pointed at a *separable sketch* that keeps the cross-time signal a full-sphere draw averages away, at cost . Which measure supplies is an experimental-design variable: the temporally optimal design gives up the spatial law, and mixing the marginal directions back in restores it at a cost linear in the mixing weight. On four pixel-based control suites under a paired protocol, prescribing the measured path law raises planning success on TwoRooms from 63.7 to 100.0, improves Reacher and Cube by more than twice the largest retraining threshold we measured, and on Push-T recovers the loss an AR(1) prior incurs against the control; prescribing no cross-time structure at all falls below the control in all four suites.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architectures (JEPA), self-supervised learning, representation learning, temporal dynamics, optimal experimental design, visual planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jECdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper points out that SIGReg, the anti-collapse regulariser in the LeWM world model, only pushes the latent at each time step toward N(0,Id)\mathcal{N}(0, I_d) and says nothing about how consecutive latents relate. As a result, a latent that is constant along each trajectory still reaches the global minimum of the LeWM loss. The paper proposes T-SIGReg, which replaces SIGReg with a target for the joint law of a window of consecutive latents. The target is N(0,Rρ,h⊗Id)\mathcal{N}(0, R_{\rho,h} \otimes I_d), where Rρ,hR_{\rho,h} is a stationary autocorrelation profile. This target is enforced with the same Epps–Pulley test on 1-D projections that SIGReg uses. The main methodological contribution is treating the choice of projection directions as an experimental design problem. Separable directions q⊗vq \otimes v make the test sensitive to errors across time without the 1/d1/d dilution of full-sphere directions.

Critical strengths
  • The diagnosis is clear and correct.
  • Treating the distribution of projection directions as an experimental design, and summarising what a design can see by the spectrum of one small matrix GμG_\mu, is a nice idea. It makes the blind spots of a sliced test computable before training.
Critical weaknesses

W1. Scope and evaluation are too narrow.

  • The method is only shown as a patch to SIGReg inside LeWM. No other world model is compared (e.g. DINO-WM, PLDM, DreamerV3, TD-MPC2). The path law is never applied to another base model, e.g. as a replacement for PLDM's VICReg-based terms. So it is unclear where the method stands among world models, or whether it helps training for anything beyond LeWM.
  • Small benchmarks, no real-world data. Four small simulated suites inherited from LeWM. OGBench appears only through single-cube manipulation, where 23–53% of episodes start with the goal already satisfied (Table 9). Only one window length is tested (h=3h = 3). No real-world data.

W2. The motivating failure is not shown to happen, and it is unclear if the gain is tied to the path law. The paper claimed a trajectory-constant latent is a global minimiser, but there's no experiment suggesting SIGReg-trained models suffer from collapse (e.g. Fig.1.). It would need further ablation study to demonstrate if the problem is real and how the proposed path regularization is help to prevent it.

W3. The writing is hard to follow. Dense prose, many coined terms ("world clock", "the ladder"), and most definitions and caveats deferred to a 35-page appendix make the main argument hard to follow.

Questions for authors
  • How does T-SIGReg compare with existing world models (e.g. DINO-WM, PLDM, TD-MPC2) under the same protocol? Can you report results under LeWM's original evaluation protocol? How does the method do on harder OGBench tasks (Visual Scene, multi-cube, Puzzle) and on real-world video?
  • Can you show a setting where a SIGReg-trained model actually ends up with a degenerate or trajectory-constant latent, and T-SIGReg prevents it?
  • Would the path law improve any base model other than LeWM?
1 comment
AuthorsOpenReview ↗

Response to Reviewer jECd

Thank you for a careful and direct review. Your concerns were fair: the original submission asked the reader to accept a failure it did not show, on a narrow evaluation, in prose that was hard to follow. We took all three seriously. The paper has been rewritten from the abstract to the conclusion, and each of your questions became a set of experiments that are now its main results. Changes that respond to your review are in pink. Numbers are planning success (%).

W1 and Questions 1 and 3: scope

Does the path law improve a base model other than LeWM? Yes. As you suggested, we replaced the VICReg-based regulariser of PLDM by T-SIGReg, and we did the same in Fast-LeWM. Mean over the four environments (Table 1):

  • LeWM: 66.3 with SIGReg, 82.5 with T-SIGReg.
  • PLDM: 64.8 with VICReg, 82.2 with T-SIGReg.
  • Fast-LeWM: 69.0 with SIGReg, 81.8 with T-SIGReg.

T-SIGReg improves each world model in each of the four environments. Table 1 also compares with Sub-JEPA, AC-MTM, TC-LeWM and temporal straightening.

DINO-WM and LeWM's original protocol. Appendix Table 9 reports every goal distance, and its shaded column is LeWM's original setting (goal 25 steps away). In that setting, T-SIGReg against SIGReg in LeWM gives 100.0 against 86.7 on TwoRooms, 82.8 against 71.8 on Cube, 89.3 against 92.2 on Push-T and 80.2 against 81.7 on Reacher. So at this short distance T-SIGReg is clearly ahead in two environments and slightly behind in two. The gain grows with the distance to the goal (Figure 5). DINO-WM, from its public checkpoints, reaches 100.0, 80.0 and 75.7 on TwoRooms, Cube and Push-T. We evaluate it only in this setting, because one planning step with it takes about 130 times longer than with LeWM (Appendix D.1).

Harder OGBench tasks. We added cube-double, cube-triple, scene and puzzle-3x3 (Appendix E, Table 15). With an AR(1) profile at ρ = 0.97, fixed before seeing any result on these tasks, T-SIGReg improves on the original regulariser of all three world models in all four tasks. For example, in LeWM: 52.4 against 40.8 on cube-double and 50.0 against 44.1 on cube-triple. With the measured profile, the configuration of Table 1, T-SIGReg is below the AR(1) profile on these tasks and below the original regulariser in three of the twelve cases. We report both rows.

Episodes that start at the goal. Without them the gain on Cube holds and grows: 87, 52 and 66 for T-SIGReg against 76, 35 and 51 for SIGReg at goals 15, 45 and 75 steps away (Appendix D.5).

Window length. We sweep it in Appendix C.5. T-SIGReg stays above SIGReg at every length on TwoRooms, Cube and Reacher. On Push-T it needs LeWM's default of three context frames and falls below SIGReg with shorter windows.

What we did not do. We do not compare with reward-based agents such as DreamerV3 and TD-MPC2, which solve a different problem (our setting has no reward), and we have no real-world video. The conclusion states both limits.

W2 and Question 2: is the collapse real, and is the gain tied to the path law?

A setting where SIGReg collapses and T-SIGReg does not (Section 5.2, Figure 1a–d). We render Reacher over a random background that is fixed within each episode. Under SIGReg the latent stops moving within an episode: only 10⁻⁵ of its variance lies within a window, against 0.36 on clean Reacher, and a linear probe no longer recovers the arm. Planning succeeds in 5.6% of episodes, below a random policy (11.3%). Under T-SIGReg the probe recovers the shoulder angle and planning succeeds in 84.9%. The other two world models show the same pattern: PLDM goes from 6.5 to 84.3 and Fast-LeWM from 7.8 to 60.5. On Cube with a natural video in the background, LeWM goes from 6.6 to 47.0 (Appendix E).

Is the gain tied to the path law? Three controls (Section 5.3). T-SIGReg prescribing no temporal structure (R = I) falls below SIGReg in all four environments. SIGReg matched to the variance T-SIGReg reaches does not improve on SIGReg. Slowness and curvature penalties help in some environments and hurt in others.

W3: writing

You were right, and we treated the writing as seriously as the experiments. The paper no longer leads with the theory. The abstract and introduction state the two problems in plain terms and show them in Figure 1. A new Section 3.1 defines the problem, the world model, the planner and SIGReg in the body, and Section 3.2 explains step by step why SIGReg allows both problems. "World clock" and "the ladder" are gone, and the body no longer depends on the appendix for definitions.

Each configuration is one training run, and we state this in the paper. We would be glad if you could look at the revised version, and if any step of the argument is still hard to follow, we would like to know and will fix it.

Reviewer_VPQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper builds on a term called SIGReg, which can prevent the collapse of the training of JEPA-style world models (this is not proposed by this paper). It proposes T-SIGReg, which constrains a window of consecutive latents to a Gaussian with a fixed temporal correlation profile. They show that random full-sphere directions dilute the temporal signal by a factor of d. Then they derive an optimal direction design, showing that this design cannot control the latent scale. They later fix it by mixing in single-time-step directions. It also improves the Success rate of control tasks.

Critical strengths
  1. The point that "a normal marginal distribution does not equal constraining the temporal structure" is explained very well, and a constructive counterexample is provided in the paper (the latent variable is constant on each trajectory). I feel like since the method is reusing the same test from SIGReg, the replacement can come with a low computational cost.

  2. I think for the 1/d dilution of full-sphere directions coming out in closed form, the experiments conducted in the paper match it quite well. Also, the blind spot of the optimal design is predicted before training, which is very good, and it does show up in the trained models. The performance gain on the three suites is significant. Also, I think the protocol is reported very transparently.

Critical weaknesses
  1. The paper compares only against SIGReg and some variant baselines of its own method. Since there are no external baselines, we don’t know whether this method is needed. One obvious comparison is missing (Temporal Straightening), which is aiming at the same problem for the same purpose. Also, it seems that the measured profile and a fixed prior give almost the same results, so any strong slowness prior might be enough.

  2. Most guarantees in the paper assume the criterion reaches zero, but I think it never happens in training. So it seems that the trained models do not match the proposed theory. Also, the trained latents have much lower variance than the target, and I didn’t see the authors separate this compression effect from the effect of the path law, so part of the gain may come from compression.

Questions for authors

Following my review above:

  1. Could the authors add a comparison with Temporal Straightening, and maybe another comparison with a baseline adding a slowness penalty (like ‖z_{t+1} − z_t‖²) on top of SIGReg?

  2. Could the authors try to separate the compression effect? For example, the authors could train a control model with a lower SIGReg weight so its latent variance matches that of T-SIGReg’s latents.

1 comment
AuthorsOpenReview ↗

Response to Reviewer VPQe

Thank you for a review that asked for specific experiments. We ran all of them, and they are now the centre of the paper. Changes that respond to your review are in green. All numbers below are planning success (%) from Table 1 unless we say otherwise.

Weakness 1 and Question 1: external baselines, temporal straightening and slowness

Temporal straightening. We compare in two ways (Table 1; details in Appendix D.3).

  • Its curvature penalty added to SIGReg in LeWM, at the two weights of their paper. At 0.01 it is level with SIGReg (mean over the four environments 66.2 against 66.3). At 0.1 it helps on TwoRooms (80.8 against 62.5) and hurts on Push-T (27.8 against 54.2).
  • Their own world model, trained with their official code: mean 46.6. Their paper covers only Push-T among our environments, so for the other three we used the settings of their closest environment. Those three are not tuned by the authors, and we say so.

T-SIGReg in LeWM reaches a mean of 82.5 and is above both in all four environments.

Slowness penalty on top of SIGReg, as you suggested, at two weights.

  • At weight 1 it nearly matches T-SIGReg on TwoRooms (98.1 against 100.0) and is level with it on Cube (75.2 against 74.7), but Push-T falls to 10.9 (SIGReg: 54.2).
  • At weight 0.1 Push-T is still below SIGReg (41.2).

So a strong slowness prior is enough where the state moves slowly, and it is harmful where the state moves fast. A linear probe shows why: under the slowness penalty the latent loses the fast-moving state of Push-T (Appendix Table 12). Slowness and curvature penalties also share a limitation: a latent that is constant along the trajectory minimises both, so neither prevents the collapse in time.

Does the profile matter? You are right that the measured profile and an AR(1) prior give similar results: on the main environments they match on all but Push-T. Prescribing no temporal structure at all (R = I) is different: it falls below SIGReg in all four, for example 31.8 against 63.7 on TwoRooms (Appendix D.4, at M = 256).

Other external baselines. Table 1 also includes Sub-JEPA, AC-MTM and TC-LeWM, and T-SIGReg replacing the regulariser of two other world models, PLDM and Fast-LeWM.

Weakness 2, first part: the guarantees assume the criterion reaches zero

We agree, and the paper now says so where the results are stated. Section 3.4 says that the theorem describes the criterion at zero, which training does not reach, and that it does not describe the trained latent. Section 4.4 says that away from zero the guarantee is only qualitative. What we can check on trained models, we check: across 22 runs, designs with a larger per-step margin, computed before training, end with a latent variance closer to the target (Appendix C.3), and Appendix Table 14 reports how far each trained latent is from the target.

Weakness 2, second part, and Question 2: separating compression from the path law

We ran two controls.

Variance. We tried what you proposed. Lowering the weight of SIGReg does not lower the variance of its latent (it stays near 1.0, Appendix Table 11), so we matched the variance directly: SIGReg is trained towards N(0, σ²I), with σ² the variance T-SIGReg reaches in each environment (0.57 to 0.93). This control does not improve on SIGReg (mean 65.5 against 66.3).

Weight. Lowering the weight of SIGReg concentrates its latent in fewer directions. On TwoRooms this does help: at weight 0.01 the mean over goal distances rises from 62.5 to 80.8. At the farthest goal it reaches 59.7, against 100.0 for T-SIGReg, and below 0.01 the latent collapses (Appendix Table 11). Our reading is that using fewer directions explains part of the gain on TwoRooms and not all of it. This sweep was run on TwoRooms only.

We hope these experiments answer your two questions, and we are glad to clarify anything else.

Reviewer_V5gEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In JEPA world models, SIGReg is the most common regulariser, avoiding representation collapse by encouraging the distributions of latent representations across the dataset to be approximately Gaussian, N(0,Id)\mathcal{N}(0, {I}_d). The problem this paper addresses is that if we're sampling from this Gaussian for multiple trajectories (i.e. sampling successive points), then every time step could get an entirely new sample from the Gaussian (so successive latents along a trajectory are entirely uncorrelated), or all trajectories could get a unique sample that then doesn't change within the trajectory. It is plausible that each of these situations could occur while still achieving a global minima in SIGReg, giving poor results.

This paper proposes an alternate regulariser, T-SIGReg, which improves temporal correlation within trajectories. T-SIGReg enforces the residual between successive latents in a trajectory to be gaussian distributed with a target correlation, which is then extended to higher dimensions and time steps by projecting each latent to a scalar and taking the weighted sum of these successive scalars. When normalised, this weighted sum is encouraged to be a unit Gaussian.

Critical strengths

1. Motivation

The paper is mostly well-motivated, explaining the shortcomings of using SIGReg as a regulariser. While I would like to see more empirical proof or a clearer visual demonstration of what the failure mode looks like, as well as significantly clearer language, the authors have detailed the issue being resolved and its importance.

2. Results The results of Figure 5 show that T-SIGReg mostly improves the planning success rate for LeWM. Whilst I would appreciate some more explanation of what the planning environments are, the disparity across different planning environments, some more visible clarity on the standard deviation between seeds and an explanation for the sign change in Push-T, the trend that T-SIGReg improves the success rate is clear.

Critical weaknesses

I was contemplating writing more points, however I only have one concern that I would need to see addressed by the authors in order to change my score:

Clarity

The paper dives into difficult technical terminology much too fast, implicitly assuming far far too much familiarity with the subject. Some examples are included below:

  • "the identifiability theory for this objective assumes exactly the cross-time structure it never enforces" (first line of the abstract)
  • use of acronyms before they're defined (e.g. UMAP, PCA-2)
  • some variables are defined for equations that they don't appear in (e.g. the line after equation 4 refers to variables M and T, which aren't used in the equation).
  • some figures aren't very informative. Figure 1 shows T-SIGReg giving a smooth plot in comparison to SIGReg, but it's unclear if that means it's necessarily better given this is just a 2-dimensional projection of a 192-dimensional representation. I can see the success rate is significantly higher for T-SIGReg, so arguably the figure should focus more on that than the smooth plots which don't seem very informative.
  • it's very difficult to interpret the motivation empirically, or understand what the failure mode looks like. While I agree latent representations of a trajectory can collapse to a constant value, it would be helpful to see the results of that visualised, or a more intuitive understanding of that be provided.
  • It's very unclear what the 4 pixel-based suites actually are, how they differ, and what they're each testing.

While individually none of these are a major issue, there are countless similar examples throughout the paper that, when combined, make it very difficult to interpret. I would highly highly encourage the authors to significantly simplify the language in order to make it more interpretable (which will make the paper more impactful). Unless this simplification happens, I cannot recommend this paper for acceptance.

Questions for authors

I thank the authors for the work they have put into the paper. I have tried to give as honest and thorough an assessment as I am capable. I have particularly struggled with the clarity of this paper, and in doing so, I have a couple of questions:

  1. Can you please explain, intuitively, what is the impact on planning when the latents of a trajectory are all identical? I'm finding it difficult to interpret what the failure mode you're solving here actually looks like.

  2. Please explain what the 4 planning environments ("pixel-based suites") are that you're evaluating against - it's not clear from the paper how they were chosen, how they differ, and what capabilities they're each testing.

1 comment
AuthorsOpenReview ↗

Response to Reviewer V5gE

Thank you for the time you put into a paper that was hard to read, and for saying exactly where it lost you. Your concern was justified. We took it as the main task of the revision: the paper has been rewritten from the abstract to the conclusion with one question in mind, whether a reader who does not know SIGReg can follow it. Changes that respond to your review are in blue.

Your examples, one by one

  • First line of the abstract. That sentence is gone. The abstract now explains what a latent world model needs, what SIGReg does, the two problems this leaves, and our fix, in that order.
  • Acronyms. UMAP and AR(1) are expanded at first use, and "PCA-2" is now "the first two principal components".
  • Variables that do not appear in the equation. Every symbol is now defined before the equation that uses it. The criterion is equation (1), and M and T both appear in it.
  • Figure 1. We rebuilt it around your suggestion. It now shows planning success against the distance to the goal (panel h) and how well latent distance follows true distance to the goal (panels i, j: correlation −0.11 under SIGReg, +0.98 under T-SIGReg). The two-dimensional plots remain only as illustration, and those of TwoRooms state the share of variance they show.
  • What the failure looks like. The top row of Figure 1 shows it in a trained model, and Section 5.2 describes it. See also our answer to your first question.
  • The four environments. Section 5.1 and the new Figure 4 describe them. See our answer to your second question.

Beyond these examples, a new Section 3.1 defines the problem, the world model, the planner and SIGReg before we build on them, and Section 3.2 explains step by step why SIGReg allows both problems.

Question 1: what happens to planning when all latents of a trajectory are identical?

The planner tries many candidate action sequences. For each one it predicts the latent the sequence would reach, and it keeps the sequence whose predicted latent is closest to the latent of the goal. If every frame of an episode has the same latent, the goal included, then every candidate ends at the same distance from the goal. The planner has nothing to choose with, and its actions are close to arbitrary. We observe this: on Reacher with a random background per episode, the model trained with SIGReg succeeds in 5.6% of episodes, below a random policy (11.3%) and far below the same model on clean Reacher (84.2%). With T-SIGReg it succeeds in 84.9%. In Figure 1c the whole episode is a single point.

A similar failure appears in a more realistic setting. On Cube with a natural video playing in the background, as in the Distracting Control Suite, the latent trained with SIGReg barely changes between consecutive frames, and planning succeeds in 6.6% of episodes, again below a random policy (9.3%). With T-SIGReg it succeeds in 47.0% (Appendix E; one training run each, in LeWM).

Question 2: what are the four environments?

They are the four pixel-based environments released with LeWM. We use them with LeWM's own datasets, evaluation code and planner, so that every comparison is made in the setting SIGReg was tuned for. Each tests a different skill: Push-T is contact-rich pushing of a T-shaped block, TwoRooms is navigation through a door, Cube is 3-D manipulation of a cube, and Reacher is long-horizon reaching with a two-joint arm. Figure 4 shows a start and a goal for each.

Two points from your comments on the results

  • Spread across seeds. Table 1 and Figure 5 now report the standard deviation over six evaluation seeds.
  • The sign change on Push-T, and the differences between environments. In the original submission T-SIGReg did not improve on SIGReg on Push-T. That was with 256 projections. With the 4096 projections we now use for every method, T-SIGReg is above SIGReg in all four environments. The gain differs between them: it is largest on TwoRooms (+37.5 points) and smallest on Push-T (+7.5, still above the retraining threshold). On TwoRooms, Reacher and Push-T the gain grows with the distance to the goal (Figure 5).

We would be glad if you could look at the new abstract, introduction and Section 3.1, which are the parts your review changed most. If something there is still unclear, we would like to know and will fix it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.