- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Semigroup-JEPA (SG-JEPA), a gravity-conditioned extension of LeWorldModel that jointly trains a visual encoder and a latent predictor using a discounted autoregressive rollout objective and SIGReg. Gravity is explicitly supplied to the model, and training uses a narrow gravity distribution while evaluation covers a substantially wider range. The paper evaluates physical-state prediction through frozen probes and downstream control through separately trained gravity-conditioned Diffusion Policies using frozen visual features. It also develops a linear feature model that separates representation closure error, predictor error, and their propagation during recursive rollout. Frozen-encoder predictor-crossover experiments are used to investigate whether the observed advantage resides in the learned representation rather than the original predictor. The reported results show improvements on several prediction and control tasks, although the gains vary across physical quantities and gravity regimes.
Critical strengths
1. A useful controlled setting for studying representation learning under changing dynamics.
The paper studies a meaningful question: whether representations learned for local prediction remain useful when their dynamics are composed over longer horizons and evaluated under changes in a physical parameter. Explicitly conditioning the model on gravity helps define the scope of the problem and cleanly separates parameter-conditioned prediction from system identification. The MuJoCo setup is also well suited to this study, as it provides access to physical-state measurements and enables complementary evaluation through both prediction and downstream control.
Several of the reported improvements are substantial. On the square dataset, the GRU variant reduces the horizon-44 position error from 2.006 m to 1.034 m compared with DINO-WM, while success on Arm Catcher Ball increases from 9.5% to 23.3%. These results suggest that the proposed training formulation has practical value beyond its conceptual motivation.
2. The encoder–predictor crossover provides a useful diagnostic beyond aggregate performance comparisons.
Freezing the learned encoders and training fresh GRU and Transformer predictors is a useful attempt to disentangle representation quality from the performance of a particular predictor checkpoint. The encoder learned with the GRU-based setup retains lower aggregate error when paired with either newly trained predictor, supporting the more limited but meaningful conclusion that the observed advantage is at least partly reflected in the learned representation itself.
The matched Transformer-conditioning experiment in Appendix D.5.1 further addresses a plausible architectural confound. Together with the paper's distinction between teacher-forced and recursive prediction, these analyses provide substantially more diagnostic insight than a comparison of final end-to-end performance alone.
Critical weaknesses
1. The specific contribution of recursive multi-step training to representation learning is not sufficiently isolated.
A central claim of the paper is that backpropagating a multi-step rollout objective through the encoder improves the learned representation. However, the main comparison with Original LeWM changes several factors simultaneously, including the predictor architecture, the training objective, and the SIGReg coefficient. In particular, the planar SG-JEPA configuration uses a SIGReg coefficient of 0.72, whereas Original LeWM uses 0.09, and Appendix D.1.3 indicates that this coefficient has a substantial effect on representation quality and downstream readout performance.
I appreciate that Appendix D.5 controls for predictor conditioning and compares different architectures under the same five-step objective. However, these experiments primarily address architecture-dependent differences in representation learning, rather than isolating the effect of recursive multi-step supervision itself. Similarly, the temporal sweep compares training horizons of 5, 10, and 20 steps, but does not include a matched one-step GRU baseline under otherwise identical settings.
The encoder–predictor crossover is useful for showing that the observed advantage persists after replacing the original predictor, but it does not identify which training intervention is responsible for that advantage. This distinction is important because the representation-shaping effect of recursive training is one of the paper's central explanatory claims. A matched one-step versus multi-step comparison, with the encoder, predictor family, regularization strength, and other training settings held fixed, would substantially strengthen this claim.
2. The relationship between OOD model selection and final evaluation should be clarified.
Appendix D.1.3 states that the SIGReg coefficient is selected using downstream rollout performance on a 5,000-episode selection cohort spanning 25 gravity values from -2 to 10. Appendix B describes a held-out planar test set with the same gravity grid and the same number of episodes. It is therefore unclear from the current description whether the selection cohort is fully episode-disjoint from the final evaluation set, including the experiments reported in Appendix D.5.
Even if the episodes are disjoint, selecting hyperparameters using physical-state rollout errors across the target gravity range still introduces access to OOD validation information. This differs from a setting in which model selection is performed entirely within the narrow source distribution and extrapolation is evaluated only afterward.
This does not by itself invalidate the zero-shot deployment setting, but it does affect how the generalization results should be interpreted. In particular, comparable access to the same OOD validation information should be provided to the baselines if hyperparameters are selected in this way. I am not assuming that test leakage occurred; rather, the current description leaves an important aspect of the evaluation protocol insufficiently specified.
Questions for authors
1. Isolating the effect of the training intervention
Could the authors provide, or point to, a comparison between one-step and five-step training under otherwise matched settings, including the same GRU architecture, encoder, gravity conditioning, SIGReg coefficient, and a comparable training budget? A five-step teacher-forced control would also help distinguish the effect of recursive feedback from the effect of simply supervising a larger number of transitions. Even a focused comparison on one representative task would substantially strengthen the claim that recursive multi-step training specifically improves the learned representation.
2. OOD model selection and data separation
What is the exact relationship between the 5,000-episode selection cohort described in Appendix D.1.3 and the evaluation cohorts used for the main results and Appendix D.5? Please clarify whether there is any episode overlap, which gravity values are available during model selection, and whether the baselines are given comparable hyperparameter-selection budgets and access to the same validation information. It would also be helpful to report which of the main conclusions continue to hold when hyperparameters are selected using only source-distribution validation data.
3. Interpretation of the probe-based diagnostic
Equation (1) defines excess physical-state error by subtracting the probe error on true encoded latents from that on predicted latents. As Eq. (73) shows, this quantity generally retains a cross term between the true-latent readout residual and the rollout-induced readout discrepancy, and therefore is not generally identical to the squared discrepancy between the two probe outputs. I appreciate that Appendix H.2 explicitly states that the teacher-forced diagnostic in Fig. 5 uses the direct same-probe comparison between the true and predicted latents, for which the true-state readout residual cancels.
Could the authors clarify which metric is used for Fig. 5(a), Fig. 5(c), and the frozen encoder–predictor crossover results in Table 6? Do the qualitative source rankings remain unchanged when reporting the direct decoded-latent discrepancy and the excess physical-state metric separately?
2 comments
final
Thank you for the additional experiments and clarifications. The matched one-step, five-step teacher-forced, and autoregressive comparisons substantially address my main concern about isolating the contribution of recursive training. The fresh-predictor experiment further strengthens the evidence that the improvement is reflected in the learned representation. I also appreciate the clarification of the two probe-based metrics.
My remaining concern is mainly about model selection. Confirming that the selection and evaluation episodes are disjoint resolves the overlap question. However, selecting hyperparameters over the broader gravity range still uses OOD validation information. It would be helpful to state this explicitly and clarify whether the baselines had comparable access to the same validation information and hyperparameter-selection budgets. This would help align the generalization claims with the evaluation setting.
Overall, the new results strengthen the paper and address the main experimental concern in my original review. I am therefore increasing my score by one point, while retaining the above concern about the evaluation protocol.
Thank you for distinguishing what the predictor crossover establishes from what requires a matched training intervention. We address the two comparisons separately and clarify which data informed model selection.
W1 / Q1 — Isolating recursive training
On 2D Square, we performed additional experiments and trained one-step, five-step teacher-forced, and five-step autoregressive models across three seeds. They use the same encoder, GRU, gravity conditioning, SIGReg weight of 0.72, training split, optimizer schedule, and 90,000 updates. The two five-step experiments supervise the same number of targets and differ in whether the predictor receives its own predictions as input. The autoregressive arm uses more predictor computation at the same update budget.
The table reports normalized physical-state MSE against ground truth, averaged over forecast horizon 1–44 (mean ± SD across training seeds). The source range is 3≤g≤5. Far OOD is g≤2 or g≥6.
Recursive training lowers far-OOD free-rollout error by 8.65% relative to one-step training and 17.22% relative to five-step teacher forcing. Both directions hold in all three seeds.
To test the role of the representation, we froze the encoders from both five-step arms and fit fresh GRUs with matched initialization and training. The autoregressively trained source retained 9.9% lower far-OOD free-rollout error (0.54563±0.02327 versus 0.60550±0.01889), with the same direction in all three source-seed pairs. This supports a representation contribution to the gain from recursive feedback at fixed supervision count.
W2 / Q2 — OOD selection and episode separation
Thank you for pointing this out. We should have made this clear in the paper. For hyperparameter selection and final evaluation, we generated separate episodes and there is no overlap. We used 2D Square as our primary task for hyperparameter sweep and reused the core training setup across the 2D and 3D tasks, including H=20, K=5, and γ=0.95, so we did not cherry-pick/tune for individual task. There is also no test leakage so we don't expect the hyperparameters to carry much information about the OOD datasets. For the baselines we also did hyperparameter tuning on 2D square, but in this case the main thing to tune is SIGReg coefficient.
Q3 — Excess physical-state error and same-probe discrepancy
We agree with the distinction in Eq. (73). Let y be the normalized physical state, q the probe output from observed latents, and p the output from predicted latents. The excess error in Eq. (1) is Δ=MSE(y,p)−MSE(y,q). The direct same-probe discrepancy is D=MSE(q,p). Their difference is the cross term 2mean[(y−q)(q−p)]. We will describe Δ as error relative to the true-latent readout baseline and use D when referring to the direct discrepancy.
Figure 5(a) and Table 6's Primary column use excess physical-state NMSE averaged over horizons 1–44. Table 6's latent MSE is in latent coordinates, and its physical-unit errors use the final forecast. Appendix H.2 defines the teacher-forced diagnostic as D. Figure 5(c) shows paired source gaps over the horizon.
In a new comparison, we computed both metrics on the same predictions across the full gravity grid. Both encoders were trained with the five-step autoregressive objective and SIGReg 0.72, then frozen and paired with a fresh GRU trained under the same protocol. Values are mean ± SD across three source-training seeds.
So the advantage persists