acceptodds
Under review as a conference paper at ICLR 2027

Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization

Abstract

Joint-Embedding Predictive Architecture (JEPA) world models learn a compact latent representation of the world that supports prediction and planning, but their capability to learn physics and generate physically realistic dynamics remains hitherto untested. In this work, we introduce SemiGroup-JEPA (SG-JEPA), which extends the LeWorldModel framework by supplying the parameter governing the physics to the temporal model via action-conditioning and jointly training an encoder and predictor through an autoregressive latent rollout. To evaluate the model's ability to generalize out of distribution, we design dynamical tasks under different gravitational fields that, despite obeying the same physical law, exhibit qualitatively different dynamics, ranging from floating motion in weak gravitational fields to rapid bouncing in strong ones. In contrast to DINO-WM, SG-JEPA reduces open-loop prediction error by up to on two-dimensional datasets, and increases control success rate up to for three-dimensional robotic datasets, for which we train independent diffusion policies. To explain this advantage, we develop a linear feature model that separates local law-conditioned error from its recursive amplification under rollout. Guided by this model, we find that back-propagating the multi-step rollout loss into the representation trains the encoder to keep the features that the predictor can carry forward, and that those are the features the dynamics depend on, so most of the gain comes from the encoder learning better features rather than from the predictor learning better dynamics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Physics-Conditioned World Models, Zero-Shot Physics Generalization, Long-Horizon Prediction, JEPA, Robot Learning, Out-of-Distribution Dynamics

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_rNwvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Semigroup-JEPA (SG-JEPA), a gravity-conditioned extension of LeWorldModel that jointly trains a visual encoder and a latent predictor using a discounted autoregressive rollout objective and SIGReg. Gravity is explicitly supplied to the model, and training uses a narrow gravity distribution while evaluation covers a substantially wider range. The paper evaluates physical-state prediction through frozen probes and downstream control through separately trained gravity-conditioned Diffusion Policies using frozen visual features. It also develops a linear feature model that separates representation closure error, predictor error, and their propagation during recursive rollout. Frozen-encoder predictor-crossover experiments are used to investigate whether the observed advantage resides in the learned representation rather than the original predictor. The reported results show improvements on several prediction and control tasks, although the gains vary across physical quantities and gravity regimes.

Critical strengths

1. A useful controlled setting for studying representation learning under changing dynamics.

The paper studies a meaningful question: whether representations learned for local prediction remain useful when their dynamics are composed over longer horizons and evaluated under changes in a physical parameter. Explicitly conditioning the model on gravity helps define the scope of the problem and cleanly separates parameter-conditioned prediction from system identification. The MuJoCo setup is also well suited to this study, as it provides access to physical-state measurements and enables complementary evaluation through both prediction and downstream control.

Several of the reported improvements are substantial. On the square dataset, the GRU variant reduces the horizon-44 position error from 2.006 m to 1.034 m compared with DINO-WM, while success on Arm Catcher Ball increases from 9.5% to 23.3%. These results suggest that the proposed training formulation has practical value beyond its conceptual motivation.

2. The encoder–predictor crossover provides a useful diagnostic beyond aggregate performance comparisons.

Freezing the learned encoders and training fresh GRU and Transformer predictors is a useful attempt to disentangle representation quality from the performance of a particular predictor checkpoint. The encoder learned with the GRU-based setup retains lower aggregate error when paired with either newly trained predictor, supporting the more limited but meaningful conclusion that the observed advantage is at least partly reflected in the learned representation itself.

The matched Transformer-conditioning experiment in Appendix D.5.1 further addresses a plausible architectural confound. Together with the paper's distinction between teacher-forced and recursive prediction, these analyses provide substantially more diagnostic insight than a comparison of final end-to-end performance alone.

Critical weaknesses

1. The specific contribution of recursive multi-step training to representation learning is not sufficiently isolated.

A central claim of the paper is that backpropagating a multi-step rollout objective through the encoder improves the learned representation. However, the main comparison with Original LeWM changes several factors simultaneously, including the predictor architecture, the training objective, and the SIGReg coefficient. In particular, the planar SG-JEPA configuration uses a SIGReg coefficient of 0.72, whereas Original LeWM uses 0.09, and Appendix D.1.3 indicates that this coefficient has a substantial effect on representation quality and downstream readout performance.

I appreciate that Appendix D.5 controls for predictor conditioning and compares different architectures under the same five-step objective. However, these experiments primarily address architecture-dependent differences in representation learning, rather than isolating the effect of recursive multi-step supervision itself. Similarly, the temporal sweep compares training horizons of 5, 10, and 20 steps, but does not include a matched one-step GRU baseline under otherwise identical settings.

The encoder–predictor crossover is useful for showing that the observed advantage persists after replacing the original predictor, but it does not identify which training intervention is responsible for that advantage. This distinction is important because the representation-shaping effect of recursive training is one of the paper's central explanatory claims. A matched one-step versus multi-step comparison, with the encoder, predictor family, regularization strength, and other training settings held fixed, would substantially strengthen this claim.

2. The relationship between OOD model selection and final evaluation should be clarified.

Appendix D.1.3 states that the SIGReg coefficient is selected using downstream rollout performance on a 5,000-episode selection cohort spanning 25 gravity values from -2 to 10. Appendix B describes a held-out planar test set with the same gravity grid and the same number of episodes. It is therefore unclear from the current description whether the selection cohort is fully episode-disjoint from the final evaluation set, including the experiments reported in Appendix D.5.

Even if the episodes are disjoint, selecting hyperparameters using physical-state rollout errors across the target gravity range still introduces access to OOD validation information. This differs from a setting in which model selection is performed entirely within the narrow source distribution and extrapolation is evaluated only afterward.

This does not by itself invalidate the zero-shot deployment setting, but it does affect how the generalization results should be interpreted. In particular, comparable access to the same OOD validation information should be provided to the baselines if hyperparameters are selected in this way. I am not assuming that test leakage occurred; rather, the current description leaves an important aspect of the evaluation protocol insufficiently specified.

Questions for authors

1. Isolating the effect of the training intervention

Could the authors provide, or point to, a comparison between one-step and five-step training under otherwise matched settings, including the same GRU architecture, encoder, gravity conditioning, SIGReg coefficient, and a comparable training budget? A five-step teacher-forced control would also help distinguish the effect of recursive feedback from the effect of simply supervising a larger number of transitions. Even a focused comparison on one representative task would substantially strengthen the claim that recursive multi-step training specifically improves the learned representation.

2. OOD model selection and data separation

What is the exact relationship between the 5,000-episode selection cohort described in Appendix D.1.3 and the evaluation cohorts used for the main results and Appendix D.5? Please clarify whether there is any episode overlap, which gravity values are available during model selection, and whether the baselines are given comparable hyperparameter-selection budgets and access to the same validation information. It would also be helpful to report which of the main conclusions continue to hold when hyperparameters are selected using only source-distribution validation data.

3. Interpretation of the probe-based diagnostic

Equation (1) defines excess physical-state error by subtracting the probe error on true encoded latents from that on predicted latents. As Eq. (73) shows, this quantity generally retains a cross term between the true-latent readout residual and the rollout-induced readout discrepancy, and therefore is not generally identical to the squared discrepancy between the two probe outputs. I appreciate that Appendix H.2 explicitly states that the teacher-forced diagnostic in Fig. 5 uses the direct same-probe comparison between the true and predicted latents, for which the true-state readout residual cancels.

Could the authors clarify which metric is used for Fig. 5(a), Fig. 5(c), and the frozen encoder–predictor crossover results in Table 6? Do the qualitative source rankings remain unchanged when reporting the direct decoded-latent discrepancy and the excess physical-state metric separately?

2 comments

Thank you for distinguishing what the predictor crossover establishes from what requires a matched training intervention. We address the two comparisons separately and clarify which data informed model selection.

W1 / Q1 — Isolating recursive training

On 2D Square, we performed additional experiments and trained one-step, five-step teacher-forced, and five-step autoregressive models across three seeds. They use the same encoder, GRU, gravity conditioning, SIGReg weight of 0.72, training split, optimizer schedule, and 90,000 updates. The two five-step experiments supervise the same number of targets and differ in whether the predictor receives its own predictions as input. The autoregressive arm uses more predictor computation at the same update budget.

The table reports normalized physical-state MSE against ground truth, averaged over forecast horizon 1–44 (mean ± SD across training seeds). The source range is 3≤g≤53\le g\le5. Far OOD is g≤2g\le2 or g≥6g\ge6.

Training objectiveFree rollout, source rangeFree rollout, far OOD
One step0.20172 ± 0.008590.53318 ± 0.01706
Five steps, teacher-forced0.19259 ± 0.001420.58836 ± 0.03334
Five steps, autoregressive0.14536 ± 0.010660.48707 ± 0.02120

Recursive training lowers far-OOD free-rollout error by 8.65% relative to one-step training and 17.22% relative to five-step teacher forcing. Both directions hold in all three seeds.

To test the role of the representation, we froze the encoders from both five-step arms and fit fresh GRUs with matched initialization and training. The autoregressively trained source retained 9.9% lower far-OOD free-rollout error (0.54563±0.023270.54563\pm0.02327 versus 0.60550±0.018890.60550\pm0.01889), with the same direction in all three source-seed pairs. This supports a representation contribution to the gain from recursive feedback at fixed supervision count.

W2 / Q2 — OOD selection and episode separation

Thank you for pointing this out. We should have made this clear in the paper. For hyperparameter selection and final evaluation, we generated separate episodes and there is no overlap. We used 2D Square as our primary task for hyperparameter sweep and reused the core training setup across the 2D and 3D tasks, including H=20H=20, K=5K=5, and γ=0.95\gamma=0.95, so we did not cherry-pick/tune for individual task. There is also no test leakage so we don't expect the hyperparameters to carry much information about the OOD datasets. For the baselines we also did hyperparameter tuning on 2D square, but in this case the main thing to tune is SIGReg coefficient.

Q3 — Excess physical-state error and same-probe discrepancy

We agree with the distinction in Eq. (73). Let yy be the normalized physical state, qq the probe output from observed latents, and pp the output from predicted latents. The excess error in Eq. (1) is Δ=MSE⁡(y,p)−MSE⁡(y,q)\Delta=\operatorname{MSE}(y,p)-\operatorname{MSE}(y,q). The direct same-probe discrepancy is D=MSE⁡(q,p)D=\operatorname{MSE}(q,p). Their difference is the cross term 2mean⁡[(y−q)(q−p)]2\operatorname{mean}[(y-q)(q-p)]. We will describe Δ\Delta as error relative to the true-latent readout baseline and use DD when referring to the direct discrepancy.

Figure 5(a) and Table 6's Primary column use excess physical-state NMSE averaged over horizons 1–44. Table 6's latent MSE is in latent coordinates, and its physical-unit errors use the final forecast. Appendix H.2 defines the teacher-forced diagnostic as DD. Figure 5(c) shows paired source gaps over the horizon.

In a new comparison, we computed both metrics on the same predictions across the full gravity grid. Both encoders were trained with the five-step autoregressive objective and SIGReg 0.72, then frozen and paired with a fresh GRU trained under the same protocol. Values are mean ± SD across three source-training seeds.

Frozen sourceDirect discrepancy DDExcess error Δ\Delta
GRU-trained encoder + fresh GRU0.36893 ± 0.020450.38697 ± 0.03600
Transformer-trained encoder + fresh GRU0.37151 ± 0.035530.39157 ± 0.02121

So the advantage persists

Reviewer_rNwvOpenReview ↗

final

Thank you for the additional experiments and clarifications. The matched one-step, five-step teacher-forced, and autoregressive comparisons substantially address my main concern about isolating the contribution of recursive training. The fresh-predictor experiment further strengthens the evidence that the improvement is reflected in the learned representation. I also appreciate the clarification of the two probe-based metrics.

My remaining concern is mainly about model selection. Confirming that the selection and evaluation episodes are disjoint resolves the overlap question. However, selecting hyperparameters over the broader gravity range still uses OOD validation information. It would be helpful to state this explicitly and clarify whether the baselines had comparable access to the same validation information and hyperparameter-selection budgets. This would help align the generalization claims with the evaluation setting.

Overall, the new results strengthen the paper and address the main experimental concern in my original review. I am therefore increasing my score by one point, while retaining the above concern about the evaluation protocol.

Reviewer_hJ57OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates whether a latent world model learned from pixels has acquired physical laws by testing long-horizon open-loop rollouts under out-of-distribution gravity values gg. The authors point out that existing JEPA-based world models either freeze the encoder, preventing the learning of dynamics-specific features, or rely on one-step losses that do not account for accumulated rollout errors. They also note that OOD generalization under changing physical parameters has not been systematically evaluated for these models. The proposed method extends an existing joint-training framework by incorporating gg as an action input and backpropagating a K-step rollout loss through the encoder, using recurrent and state-space models as predictors. In simulated environments with controlled gravity, the method outperforms the baselines in both probe-based prediction errors and Diffusion Policy control success rates. Through crossover experiments and linear model analysis, the authors argue that the improvements come from the encoder representations rather than the predictor.

Critical strengths
  1. Alignment between the research question and experimental design. The paper operationalizes the question of whether a world model has learned physical laws along two dimensions: (i) whether physical states remain consistent during open-loop rollouts much longer than the rollout horizon used during training, and (ii) whether this consistency holds across a broad grid of gravity values outside the narrow gravity distribution sampled during training. By testing both dimensions simultaneously while keeping conditions other than gravity fixed, the experimental design aims to attribute performance differences specifically to the abilities to compose and extrapolate physical laws.

  2. Experiments isolating the source of improvement. Through crossover experiments that freeze the learned encoder and train a new predictor, the authors control for the predictor and show that the encoder jointly trained with the GRU predictor outperforms the one trained with the Transformer predictor for both fresh predictors tested. Furthermore, in one-step comparisons using teacher forcing to eliminate rollout effects, the SG-JEPA encoder has lower one-step error across all gravity values, with a substantial gap in far-OOD settings. This shows that the improvement already exists in representation quality before error accumulation occurs. The authors also show that, for the same pair of encoders, the free-rollout performance gap is several times larger than the one-step gap.

Critical weaknesses
  1. Limited methodological contribution. Multi-step rollout loss (Venkatraman, 2015; Lambert, 2021), parameter-conditioned inputs (the CaDM family), and joint encoder training (LeWM) are all existing techniques. Since gg is directly provided rather than inferred, the setting is easier than those considered in prior work, as the authors themselves acknowledge in Appendix A (“SG-JEPA removes system identification from the experiment”).
  2. The experimental findings are largely predictable from the design. Although the experiments clearly isolate the source of improvement, their conclusions are not novel. If the rollout loss is backpropagated through the encoder, the encoder changes, and if the encoder changes, performance differences can persist regardless of what predictor is placed on top of it. This follows directly from the design. Similarly, the propagation of one-step differences through rollouts follows from the recursion itself. The remaining results—reduced rollout error when the model is trained on the multi-step rollout objective it is later evaluated on, confirmation that the model uses the gg input (D.4), the superiority of a task-trained encoder over frozen DINOv2, and the effects of post-training (Appendix G)—are also predictable in advance. The questions the experiments do not answer are why GRU-based joint training produces a better encoder and what additional information that encoder captures. These are the questions that would be most interesting to investigate, but the paper only partially addresses them in Appendix D.5 and does not resolve them.
Questions for authors
  1. Mismatch between the training objective and evaluation metrics. SG-JEPA is trained using latent rollout MSE. However, in the crossover experiments in Table 6, the Transformer-joint encoder achieves lower latent MSE than the GRU-joint encoder but higher physical errors, with the difference concentrated in velocity. Similarly, in Section D.1.1, the configuration with the lowest validation loss shows poor probe performance. This indicates that the optimized objective does not consistently track the evaluation metrics. If so, through what mechanism does the GRU-joint encoder achieve its improvements beyond minimizing latent distance? Applying linear probes to the encoded latents of both encoders before rollout and comparing their ability to predict position, velocity, and angular velocity would help identify which components of the representations account for the improvement.
  2. Necessity of backpropagating rollout loss through the encoder. The individual components of this paper—multi-step rollout loss, parameter-conditioned inputs, and joint encoder training—have all been explored in prior work. The new contribution is their combination, specifically backpropagating the rollout loss through the encoder. However, the necessity of this combination has not been validated. Are there results comparing SG-JEPA with a variant that freezes a LeWM encoder pretrained with the one-step objective and trains only the predictor with the multi-step rollout loss? If this variant performs comparably to SG-JEPA, backpropagating through the encoder would be unnecessary, and the claim in Section 4 that the OOD advantage lies in the representation the encoder learns would require further explanation. Conversely, if this variant performs substantially worse, it would provide experimental support for the necessity of the proposed combination. The current crossover experiments only show that the two learned encoders differ; they do not show what happens when gradients to the encoder are blocked.
6 comments

Thank you for the thoughtful review. We answer the concerns in the following and hope the reviewer can clarify about the requested experiments in Q2.

W1 / W2 — What SG-JEPA adds and why the result was uncertain

We acknowledge that some ingredients have precedents. DaD addresses multi-step error in learned time series, Lambert et al. predict future physical states at a specified horizon, and CaDM infers a dynamics context from recent transitions. But it's not obviously true the same ideas will work or train stably under the self-supervised learning paradigm, especially with the new training loss + SIGReg regularizer. We deliberately design the architecture to be as simple as possible instead of forcing some rule/dynamics; and hope the model will figure things out by learning. Also this way of thinking also has the most potential for it to be scalable/generalizable later. However, naturally if the model looks simple, the ingredients will have precedents, but this is more of post-hoc realization. We did not expect it to work a priori.

Moreover for the use case of physics generalization, generalizing to different gravities, we do not find any precedents. We believe it will be a useful contribution for the community to initiate this discussion about robotic foundation models beyond Earth gravity, in order to deploy robots to the Moon or Mars. Our work sets the cornerstone and shows that there exists a model that has the potential to generalize beyond Earth environment. Of course VLA or WAM might also have the potential, which are waiting to be tested out.

And indeed gravity g is supplied as action here and we show in Fig 14 it does use this value. We mention in limitation that we will work on models that infer physics directly.

Q1 — Latent MSE and the information available before rollout

Yes training objective here does not necessarily correlates with downstream performance, which is a common issue for self-supervised learning models such as VJEPA. And we never attempt to make such a claim in the work, all of our headline results use downstream performance. Note that even in LLM there is no consensus that lower loss will lead to better downstream performance (e.g. https://arxiv.org/abs/2210.14199, https://arxiv.org/abs/2507.00885). Nevertheless one cannot deny usefulness of these models. We showed that the encoder learns better dynamics with GRU predictor. This is justified in Fig 5 with downstream performance

Q2

Can the reviewer clarify what extra experiments you'd like us to do here? In previous VJEPA works people used stop-gradient to stop backpropagating through the encoder. The main purpose of them doing this is to avoid representation collapse during training. We deliberately used SIGReg regularizer to remove this.

Reviewer_hJ57OpenReview ↗

Clarification of the additional experiment

To separate the predictor’s and the encoder’s contributions, please freeze the K=1 and K=5 encoders already trained (3 seeds) and, on each, train a fresh predictor with the K=1 and the K=5 autoregressive objective, giving a 2×2 grid. Reporting free-rollout error (ID and far-OOD) for all four cells would show how much of the gain comes from training the predictor with the rollout loss versus from the encoder having been trained with it. The {K=1 encoder, K=5 predictor} cell is the one currently missing and the one that distinguishes this work from prior predictor-only rollout training.

Thank you for suggesting this comparison. We froze the K=1K=1 and K=5K=5 encoders and trained fresh GRU predictors with each objective on both frozen encoders. The table reports free-rollout physical-state NMSE (mean ± SD across three encoder-training seeds).

Frozen encoderFresh predictorIDFar OOD
K=1K=1K=1K=10.1718 ± 0.01010.5927 ± 0.0327
K=1K=1K=5K=5 autoregressive0.1315 ± 0.00260.5361 ± 0.0512
K=5K=5K=1K=10.1836 ± 0.00300.5506 ± 0.0167
K=5K=5K=5K=5 autoregressive0.1273 ± 0.00180.5456 ± 0.0233

With the one-step encoder frozen, five-step predictor training reduced mean rollout error by 23.5% ID and 9.5% far OOD. With a one-step predictor, switching to the five-step encoder reduced mean far-OOD error from 0.5927 to 0.5506. With a five-step predictor, the two encoders yielded similar far-OOD means (0.5361 and 0.5456).

We will revise our interpretation accordingly. Predictor-only rollout training improves far-OOD forecasting on Square, and the effect of the encoder's training objective depends on how the fresh predictor is trained. The one-step encoder with a five-step predictor has the lowest far-OOD mean.

We appreciate if the reviewer has any more concerns so we can address them before the rebuttal period closes.

Reviewer_EqcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SG-JEPA, a gravity-conditioned extension of LeWorldModel trained through discounted autoregressive latent rollouts. It evaluates physical-state prediction and downstream control in MuJoCo environments with training gravities concentrated in a narrow range and evaluation gravities spanning a wider range. A linear feature model separates local prediction defects from their propagation under repeated updates. Encoder–predictor crossover and teacher-forcing/free-rollout experiments investigate where the observed advantage resides.

Critical strengths
  1. Section 4 makes a useful distinction between local prediction quality and recursive error propagation. Importantly, Figure 5(b) provides evidence of an advantage before recursive feedback: with matched fresh predictors, SG-JEPA representations yield lower teacher-forced local-error point estimates across the tested gravities, including an approximately 32% reduction on the selected far-OOD, collision-free transitions.
  2. The crossover experiments, gravity-conditioning interventions, and additional conditioning/architecture ablations go beyond reporting aggregate performance. The theoretical appendix also explicitly acknowledges several limitations of the surrogate analysis.
Critical weaknesses
  1. The experiments do not isolate the contribution of multi-step training to OOD generalization. Figure 5(b) reports lower local OOD error than frozen DINO-WM features, but omits Original LeWM, which already jointly trains its encoder and predictor. Matched fresh predictors do not control this difference in encoder training. Figure 5(a) instead varies predictor architecture while using multi-step objectives. Thus, the results establish a representation-associated advantage, but do not establish that the proposed rollout objective causes it. This is consequential because that causal connection is central to the paper’s explanation of its gains. A matched one-step versus multi-step comparison is needed.

  2. The contribution beyond established multi-step latent learning is insufficiently distinguished. Recursive latent prediction and multi-step consistency supervision have substantial precedent, notably in PlaNet and TD-MPC. The paper does not clearly establish what additional principle its gravity-conditioned objective introduces, while the semigroup composition identity follows from repeatedly applying a shared update. The controlled evaluation and representation diagnostics are useful, but the methodological distinction and additional scientific insight remain insufficiently established. This limits the significance of the contribution, particularly given the attribution gap above.

References:

Questions for authors
  1. Choice of comparator in Figure 5(b). Could you include Original LeWM representations in the same teacher-forced local-error evaluation, using matched fresh predictors? How much of the reported far-OOD improvement remains relative to this jointly trained JEPA baseline, rather than frozen DINO-WM features?

  2. Effect of the multi-step objective. With architecture, conditioning, regularization, and training protocol held fixed, how does K=1 compare with K>1? Please report teacher-forced local errors and free-rollout errors separately for ID and OOD gravities. This would help distinguish improvements in local extrapolation from improvements that emerge during recursive prediction.

  3. Distinction from prior work. What specifically distinguishes the proposed formulation from established multi-step latent-consistency approaches such as PlaNet and TD-MPC? Which result most directly establishes the additional methodological or scientific contribution beyond applying these principles within the LeWM framework?

1 comment

SG-JEPA reviewer responses

Thank you for pointing out the connection to prior rollout objectives and for proposing the matched comparisons. We will clarify the scope of the contribution and distinguish the evidence about the learned representation from the effect of the training objective.

W1 / Q2 — Matched one-step versus multi-step training

We agree that Figure 5 does not isolate the rollout objective. We have now completed a matched comparison on the 2D square task across 3 training seeds 42, 43, and 44.

The one-step case (K=1K=1) and the autoregressive case (K=5K=5) share the architecture, conditioning, SIGReg coefficient, initialization within each seed. The teacher-forced five-step case (K=5K=5) tests the effect of supervising more targets while feeding true latents as context.

We report normalized physical-state MSE against ground truth at epoch 20. We average over evaluation horizon 1–44 and all transitions and report mean ± training-seed SD. The ID cohort contains 1,000 episodes at 3≤g≤53\le g\le 5. The OOD cohort contains 4,000 episodes at g<3g<3 or g>5g>5.

EvaluationGravitiesK=1K=1K=5K=5 autoregressiveK=5K=5 teacher-forced
Teacher-forced localID0.0341 ± 0.00280.0312 ± 0.00300.0362 ± 0.0011
Teacher-forced localOOD0.0454 ± 0.00370.0614 ± 0.00630.0529 ± 0.0015
Free rolloutID0.2017 ± 0.00860.1454 ± 0.01070.1926 ± 0.0014
Free rolloutOOD0.4986 ± 0.01530.4529 ± 0.01800.5496 ± 0.0299

Compared with K=1K=1, autoregressive K=5K=5 reduces free-rollout error by 27.9% ID and 9.2% OOD. Local error falls by 8.4% ID and rises by 35.2% OOD. We note that in practice free-rollout error is what we care the most about because in simulation we know position of ball exactly but in reality we won't have access to the exact next-frame position/velocity. Teacher-forced K=5K=5 raises OOD rollout error by 10.2% relative to K=1K=1, so supervising five targets alone does not account for the observed gain. Under the earlier far-OOD definition (g≤2g\le 2 or g≥6g\ge 6), autoregressive training raises local error from 0.0487 to 0.0665 and lowers free-rollout error from 0.5332 to 0.4871.

W2 / Q3 — Distinction from prior work and scientific contribution

We agree that PlaNet and TD-MPC established precedents for multi-step latent supervision in model-based reinforcement learning. PlaNet uses variational overshooting in a generative world model. TD-MPC combines rollout consistency against a moving-average encoder with reward and value losses. However it's not obviously true why such method will be beneficial for JEPA-type world model, in particular with the new regularizer SIGReg. To our knowledge no prior work has shown this. SG-JEPA not only replaces LeWorldModel's one-step loss with recursive rollout loss, we also showed that when the predictor is GRU or state-space model, we actually learn dynamics better. Moreover, the main interesting results lie in the physics generalization. To our knowledge no prior work considered gravity generation of world model (trained on a small set of gravity values, test and have a model of good generalization).

Q1 — Original LeWM in the teacher-forced comparison

As per reviewer's request, we add Original LeWM to Fig 5(b) comparison. We evaluated teacher-forced predictions on a strict subset with no collisions in either the context or the forecast transition.

The values below are means over seeds 42, 43, and 44. DD is the same-probe discrepancy, and EE is normalized physical error against ground truth. Across far-OOD gravities (g≤2g\le 2 or g≥6g\ge 6), the three-seed averages are

ModelSame-probe discrepancy DDGround-truth error EE
Original LeWM0.084150.11138
SG-JEPA0.020940.04904

In this comparison, SG-JEPA has 75.1% lower DD and 56.0% lower EE than Original LeWM. The following provides values for some specific gravity values

Gravity ggOriginal LeWM DDOriginal LeWM EESG-JEPA DDSG-JEPA EE
−20.024120.051820.113960.16830
40.000640.007730.000430.00966
80.147640.182890.004320.03047
100.394570.486260.026030.08561

The DINO-WM gives a mean of 0.1077 across 18 far-OOD gravity points.

Reviewer_SvRxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Semigroup-JEPA, extending JEPA-based world models through gravity conditioning and multistep latent rollout training. They evaluate generalization to gravitational conditions outside the training distribution using simulated physical dynamics and robotic control tasks. A theoretical analysis and encoder–predictor crossover experiments examine whether the improvement comes from the learned representations or the transition predictor.

Critical strengths

I found the analysis of where the improvement comes from particularly interesting. The encoder–predictor crossover experiments provide useful evidence that the advantage is primarily associated with the learned representation rather than simply the predictor architecture.

The evaluation is reasonably broad, covering long-horizon prediction, different gravity values, and downstream control. The improvements are meaningful, particularly under distribution shifts, although the absolute control success rates remain relatively modest.

Critical weaknesses

The experimental comparisons do not fully isolate the proposed training objective. The method changes multiple components, including the predictor architecture and rollout training. A direct comparison against an otherwise identical GRU trained with a one-step objective would make the contribution considerably clearer.

World-model training uses a single random seed. The reported uncertainty across evaluation episodes therefore does not establish how consistent the improvements are across independently trained models.

Questions for authors

Can the authors compare one-step and multistep training using the same GRU architecture, gravity conditioning, and optimization settings, preferably across several training seeds? This would help determine how much of the improvement is specifically due to the proposed rollout objective.

1 comment

Thank you for finding our work interesting.

For your suggested comparison, we did further experiments on 2D Square dataset, we trained three models per objective with the same encoder and GRU architectures, gravity conditioning, SIGReg coefficient, data and optimizer schedule per run. We paired initializations across objectives within each seed. The table reports free-rollout physical-state NMSE averaged over forecast steps 1–44, as mean ± SD across three seeds (lower is better)

Training objectiveSource range (3≤g≤53\le g\le5)Far OOD (g≤2g\le2 or g≥6g\ge6)
One step0.20172 ± 0.008590.53318 ± 0.01706
Five steps, teacher-forced0.19259 ± 0.001420.58836 ± 0.03334
Five steps, autoregressive0.14536 ± 0.010660.48707 ± 0.02120

Five-step autoregressive training lowered far-OOD rollout error relative to both controls in all three paired training seeds. Across seeds, mean error fell by 8.65% relative to one-step training and 17.22% relative to five-step teacher forcing.

Reviewer_DM9LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Semigroup-JEPA (SG-JEPA), a gravity-conditioned extension of LeWorldModel that jointly trains an encoder and a predictor through multi-step autoregressive latent rollouts. The method is evaluated on simulated physical prediction and robotic control tasks, with particular attention to gravity values outside the main training distribution and prediction horizons longer than those used during training. The paper also develops a linear feature model to analyze local prediction errors and their propagation, and uses encoder–predictor crossover experiments to investigate the source of the empirical gains.

Critical strengths
  1. A relevant research question and a controlled experimental setting. Investigating whether latent world models remain useful under changing physical conditions and extended prediction horizons is worthwhile. Varying gravity provides a tractable setting for studying these questions, and the evaluations cover both physical-state prediction and downstream control.

  2. Experiments that go beyond aggregate performance comparisons. The encoder–predictor crossover study is particularly interesting: the advantage persists when the original predictor is replaced, supporting the interpretation that useful information is retained in the learned representation. The counterfactual gravity-conditioning experiment further checks whether the predictor actually uses the supplied gravity.

  3. A useful connection between analysis and empirical evaluation. The distinction between local prediction error and its propagation through recursive rollout provides a helpful framework for interpreting long-horizon behavior. The supplementary material also acknowledges the assumptions and limitations of the theoretical model.

Critical weaknesses
  1. The motivation for gravity-conditioned prediction and the scope of “new environments” should be clearer.

The Introduction motivates the work through the broad objective of generalizing to out-of-distribution dynamics. However, the transition from this objective to gravity-conditioned prediction is not sufficiently developed. The Earth-to-Mars example provides intuition, but does not fully explain why gravity variation is an appropriate controlled test of the intended capability, or why gravity is explicitly supplied to the model.

Appendix A provides a clearer rationale: the scene and control interface remain fixed, gravity is given, and the setting isolates interpolation and extrapolation along a known physical coordinate. This explanation would be valuable in the Introduction.

Could the authors clarify whether “adapting to new environments” primarily refers to using a single conditional model across different, explicitly provided gravity values? It would help to distinguish this setting from identifying unknown environmental parameters or adapting to broader changes in dynamics.

2.Minor Comments on Presentation

(a). The rollout-loss equation around lines 145–147 is central to the method but is unnumbered. Please consider numbering it to facilitate references in the discussion and supplementary material.

(b). The two green colors used for the GRU and SSM variants in Fig. 3 are difficult to distinguish. More contrasting colors, or additional hatching, would improve readability and accessibility.

(c). Figure 3 normalizes each method’s error by the corresponding DINO-WM error. Consequently, all DINO-WM bars are fixed at 1 and provide no additional numerical information. A horizontal reference line at (y=1) could replace these bars and reduce visual redundancy. The absolute DINO-WM errors below each group should be retained, since they provide useful information about the underlying error scale.

Questions for authors
  1. The interpretation of the probe-adjusted error requires clarification. Section 2.3 states that subtracting the probe error on encoded ground-truth latents “isolates the error introduced by the rollout.” Could the authors explain precisely in what sense this isolation holds?

  2. The gravity-specific performance reversals in Fig. 4 deserve further analysis. Figures 4(d)–(f) show that DINO-WM outperforms SG-JEPA at some gravity values. Could the authors analyze representative failure cases and discuss whether the failures originate from the visual representation, the policy’s response to gravity conditioning, or contact-sensitive action errors? Diagnostics such as contact timing, paddle orientation, and post-impact ball velocity would be informative for the basket task.

3 comments

Thank you very much for your careful readthrough and constructive feedbacks on the details, writing and presentation. We will work on these points and update the manuscript together with other rebuttal points. We also appreciate that you find our work useful and interesting. In the following we will address the points that need clarification and further investigation. We will add these to the manuscript.

W1 — Scope of gravity generalization

Yes "adapting to new environments" means a single model with fixed weights and supplied gravity value. Even so the model still has to learn from pixels how trajectories change with gravity and carry those predictions forward over rollouts. The same methodology will work for other parameters or multiple parameters, but inferring unknown environmental parameters and adapting to it will be next level of difficulty. And it's the direction we are aiming for in followup works. We will make these clear in our writing.

Q1 — Interpretation of probe-adjusted error

By "probe-adjusted error" we mean that the final physical error consists of several parts, one of them is the intrinsic readout error introduced by the probe. And the rest is compounded rollout error. To do this, we score the same physical states twice with the same frozen probe on the same evaluation samples. In one pass the probe reads latents from the rollout. In the other it reads latents from the observed frames. The second pass measures the readout error already present with observed inputs. Subtracting it measures how much physical-state MSE changes when we feed the probe rollout latents.

Q2 — What drives the basket-task reversals?

We appreciate the reviewer for pointing this out and our further investigation and experiments actually strengthened our results. The most noticeable ones that have mixed outcomes are Franka Basket, where we not only require the model to understand ball dynamics, but also manipulate Franka arm to control paddle to hit the ball to enter the basket. Note that we intentionally design this very difficult task and low success rate is expected so that it will serve as a challenging benchmark for future models.

We did a thorough investigation of this task generation and found that the problem lies in the evaluation set generation. In each episode we are supposed to vary the initial position and velocity of the ball while keeping basket and arm position fixed. When generating each episode we impose a criteria that the task must be "achievable", meaning that there is a way the ball will enter the basket, otherwise it will fall back to some achievable configuration. However this is very hard for larger gravity values, imagine we have g=16, unless the paddle hits very hard, the ball will almost always fall short of the basket. This results in the generation always falling back to certain configurations and thus lacking variety. Some of these configurations always work with Dino-wm and always fail with SG-JEPA, giving the points in Fig 4(e) where one method is high and the other is almost 0 success rate.

We increased variety by adding perturbation to the ball initial position and velocity, and generated a new evaluation set covering 25 gravity values. The evaluation comprises 90,000 rollouts across two fixed policies. Each method, gravity, and condition has 600 rollouts from 200 parent episode IDs and three seeds. The table shows SG-JEPA / DINO-WM strict basket-entry rates for representative gravities and the results actually strengthen our previous observation. We will also update the figure in paper.

Gravity (m/s²)OriginalPosition ±2 mm, velocity ±0.5%Position ±1 cm, velocity ±2.5%
5.543.7% / 27.8%36.3% / 28.2%37.2% / 31.5%
8.8744.0% / 42.7%34.0% / 31.2%32.3% / 27.0%
9.739.0% / 41.0%31.3% / 20.7%28.2% / 26.2%
13.92.0% / 61.2%43.3% / 39.5%39.7% / 28.5%
16.058.2% / 0.8%22.8% / 20.7%20.3% / 13.8%
Mean over 25 gravities30.36% / 27.60%24.53% / 19.96%23.02% / 19.37%

We further investigate the failure cases more carefully. We replayed recorded action sequences on both original and perturbed starts to distinguish changes in contact geometry from changes in the policy's actions. Two selected cases illustrate the difference.

  • SG-JEPA at g=7.6g=7.6. Replaying the same actions, the small ball perturbation turned a basket entry into a miss. The third floor bounce occurred about 41 ms later, and the ball struck closer to the paddle's lower edge even though the paddle tilt was nearly identical and both contacts fell in the same recorded frame. The outgoing vertical velocity changed from +2.58+2.58 to −0.65-0.65 m/s. This contact sequence was sensitive to the simulator time step. The original start succeeded at the benchmark step size and missed at both finer step sizes.
  • DINO-WM at g=13.9g=13.9. The original actions still produced a basket entry from the perturbed start. The actions generated by the policy from that same perturbed start led to a miss. The second strike occurred about 35 ms later, and the outgoing velocity (vx,vz)(v_x,v_z) shifted from (−1.99,3.49)(-1.99,3.49) to (−3.09,3.96)(-3.09,3.96) m/s. Both the simulator gravity and the supplied gravity were held fixed, and the replay changed the full action sequence.

The policy's response also rescued some perturbed starts. At g=9.7g=9.7, for example, SG-JEPA missed when replaying the original actions and succeeded with actions generated from the perturbed start. These cases show how a small change in the ball's initial state can alter the contact geometry and how the policy's response can help or hurt.

The clarification regarding W1 adequately addresses my concern about the scope of gravity generalization. For Q1, the two-pass evaluation procedure is clearer, but subtracting the ground-truth-latent probe MSE does not strictly isolate rollout error, as the difference also contains a cross term involving the original probe error. I suggest describing this metric more precisely as the excess physical-state MSE relative to the ground-truth-latent baseline.

For Q2, the additional perturbation experiments and failure-case analyses provide a reasonable explanation for the gravity-specific performance reversals. The results show that SG-JEPA maintains an average advantage under perturbations, but also reveal substantial sensitivity to initial conditions at certain gravity values (e.g., g = 13.9 and 16.0). This sensitivity should be acknowledged in the revised manuscript. One remaining question is whether the perturbed initial states still satisfy the original "achievable" criterion. Overall, my main concerns have been sufficiently addressed, and the remaining issues can be resolved through clarification and revision. I will maintain my rating.

Comments on the paper

3 comments

Thanks for the comments.

We hope to clarify that free rollout loss is the main metric we are considering in the paper. It measures what happens when the model predicts next steps using its own latents. Teacher-forced local error, which you point out as problematic, is only in Fig 5(b) as a sanity check and to isolate confounds when doing analysis. We NEVER use it anywhere except Fig 5(b) and it's not representative of the model capability, we should look at free rollout loss mainly. As seen in provided extra experiments, our method is most robust across compared methods.

As for the concern about asymmetric OOD generalization, we remain honest and faithful to the simulated results and not cherry-pick, Fig 18, 19, 24, 25 give more comprehensive overview about OOD generalization at specific gravity values. As can be seen, for most values our method is doing better and g=-2 is an extreme and never happens in reality (instead of freefall it's doing upward acceleration), so should not be taken as a representative failure case. The hyperparameter selection dataset does not have overlap with evaluation dataset.

For methodological novelty, the two papers the reviewer cites are indeed precedents for multi-step latent prediction and we acknowledge that, will cite in revised version, but the setup and use cases are totally different. We've also further clarified our main contribution in our rebuttal above. Could the reviewer point to specific evidence of other closely related works that consider similar setup or use case? Or is there anything we can clarify further?

Just to add on to the novelty part,

We acknowledge that some ingredients have precedents. But it's not obviously true the same ideas will work or train stably under the self-supervised learning paradigm, especially with the new training loss + SIGReg regularizer. We deliberately design the architecture to be as simple as possible instead of forcing some rule/dynamics; and hope the model will figure things out by learning. Also this way of thinking also has the most potential for it to be scalable/generalizable later. However, naturally if the model looks simple, the ingredients will have precedents, but this is more of post-hoc realization. We did not expect it to work a priori.

Moreover for the use case of physics generalization, generalizing to different gravities, we do not find any precedents. We believe it will be a useful contribution for the community to initiate this discussion about robotic foundation models beyond Earth gravity, in order to deploy robots to the Moon or Mars. Our work sets the cornerstone and shows that there exists a model that has the potential to generalize beyond Earth environment. Of course VLA or WAM might also have the potential, which are waiting to be tested out.

Let us know if you have further concerns if any further experiments you'd like us to provide.

Thanks for the clarification. I agree that free-rollout performance is the primary metric, and the additional experiments provide useful evidence of improvement. However, my concern about local error relates specifically to the paper's mechanistic claim that multi-step training improves the learned representations. The matched ablation does not clearly support this explanation.

Regarding hyperparameter selection, my concern is not episode overlap, but whether OOD gravity performance was used to select hyperparameters. Appendix D.1.3 explicitly states that SIGReg was selected based on rollout performance across g∈[−2,10]g\in[-2,10]. Could you clarify whether this constitutes OOD-based model selection?

On novelty, I acknowledge the value of the gravity-generalization evaluation. My concern is primarily methodological: the proposed training objective closely follows established autoregressive rollout learning, and the current evidence does not establish a sufficiently distinct contribution. I appreciate the additional experiments, but these concerns remain, so I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.