- Rating
- 2(+1)
- Confidence
- 3(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes G-JEPA, an end-to-end JEPA world model that augments latent prediction with physical-state supervision. A shared causal S4D module aggregates temporal context over both the observed and the predicted latent sequences and regresses the corresponding physical states. For planning, it trains a latent goal-conditioned diffusion action prior and mixes its samples with Gaussian CEM proposals under a shared world-model scoring loop. Experiments on PushT, OGBench Cube, and TwoRooms report higher planning success than the retrained baselines on the manipulation tasks, with ablations over the grounding losses, temporal architecture, proposal mixture ratio, and action-selection strategy.
Critical strengths
The paper shows a substantial and consistent improvement over the evaluated baselines on PushT, with or without the diffusion prior, and supports this with a fairly complete set of ablations over the grounding losses, the temporal architecture, the action-selection strategy, and the proposal ratio. This establishes that the system works and makes it possible to trace where the gains come from.
Critical weaknesses
-
The contribution is incremental, and the paper’s own ablations do not show that the new parts are responsible for the gains. Supervising JEPA latents with physical states was already introduced by PhyLatent and PSG-JEPA. The photometric invariance loss is a standard self-supervised objective that PhyLatent also uses. Mixing policy samples into Gaussian CEM is the planner used in TD-MPC. The only new elements are therefore the temporal S4D module and the grounding of predicted latents. However, the ablation in Table 3 adds the losses cumulatively, so the invariance loss is never evaluated on its own. On PushT, state grounding alone does not improve over LeWM, the large gain appears only when invariance is added, and grounding the predicted latents adds nothing further. The temporal module itself is never compared against a per-frame grounding head. It is therefore not possible to tell from the reported experiments whether the proposed grounding contributes anything beyond what the invariance loss alone would give. On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy. This is consequential because the method’s main results are not shown to depend on its new components.
-
The chosen tasks do not require temporal context, so the central claim cannot be tested on them. The motivation for temporal grounding is that the physical state may need to be inferred from evidence distributed across several frames. However, PushT and OGBench Cube are both rendered from a single fixed viewpoint, and the physical state is almost entirely visible in each frame. Apart from the PushT agent velocity and transient occlusion of the cube by the gripper, every grounding target can be recovered from the current image. On such tasks, a module that aggregates history has little to add, and the paper reports no setting with partial observability or ambiguous single frames where it would. Moreover, Appendix A.2 states that training uses four-frame windows, so the grounding module never aggregates more than three observed frames. The temporal context available to it is therefore very short. The paper also reports no history-length ablation and no physical-state prediction error, so there is no direct evidence that the S4D module uses temporal context at all. In addition, the module is discarded at planning time. Its effect on planning can therefore only be indirect, through the representation it shapes during training, and this effect is never measured. This is consequential because the experimental design leaves the main claim without any test that could have falsified it.
Questions for authors
-
What direct evidence shows that temporal aggregation contributes to the results? The grounding module sees at most three observed frames during training (Appendix A.2), and PushT and Cube are almost fully observable from a single frame. Can the authors report (a) success with the S4D module replaced by a per-frame grounding head of matched capacity, and (b) success as a function of training context length, for example 1, 3, 8, and 16 frames?
-
Can the authors show results on a setting where history is actually needed, for example the same tasks with partial occlusion of the block or cube at varying levels? This is where the claimed advantage should appear, and no such setting is evaluated.
-
In Table 1, PSG-JEPA performs far below the ungrounded LeWM on PushT-Ada, even though it adds physical-state supervision. Since the PushT configuration for PSG-JEPA was added by the authors, can they clarify how it was implemented, including the state targets used and whether its hyperparameters were tuned, and explain why a grounded model would fall this far below an ungrounded one?
10 comments
W1-2: However, the ablation in Table 3 adds the losses cumulatively, so the invariance loss is never evaluated on its own. On PushT, state grounding alone does not improve over LeWM, the large gain appears only when invariance is added, and grounding the predicted latents adds nothing further.
Response to W1-2: We thank the reviewer for pointing out the missing isolated ablation. We have added an invariance-only setting and extended the evaluation to two additional challenging task sets to examine the individual and combined effects of these components. All settings use Gaussian CEM without an action prior. As shown in Table 4, neither state supervision nor invariance alone consistently improves over LeWM. Combining them improves over either component alone on all datasets. We agree that adding predicted-state supervision does not increase mean success on PushT-Ada. It improves mean success on Cube and both challenging task sets. These results support the contribution of predicted-state supervision in several settings, with larger improvements on the challenging task sets.
Table 4. Component ablations of temporal physical grounding and nuisance invariance.
| State (S) | Invariance (I) | Predicted state (P) | PushT-Ada (%) | OGBench Cube (%) | PushT-Challenging(25 steps) (%) | Cube-Challenging(25 steps) (%) |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 56.90 ± 2.35 | 72.80 ± 2.53 | 80.69 ± 1.65 | 15.00 ± 2.65 |
| ✓ | ✗ | ✗ | 55.73 ± 1.58 | 75.40 ± 1.65 | 76.98 ± 0.79 | 26.00 ± 1.00 |
| ✗ | ✓ | ✗ | 19.40 ± 1.26 | 74.20 ± 1.75 | 51.59 ± 2.38 | 19.33 ± 5.13 |
| ✓ | ✓ | ✗ | 79.43 ± 0.90 | 76.20 ± 2.57 | 88.89 ± 2.38 | 26.67 ± 2.89 |
| ✓ | ✓ | ✓ | 79.43 ± 1.58 | 78.60 ± 4.12 | 92.06 ± 2.38 | 33.00 ± 1.73 |
W1-3: The temporal module itself is never compared against a per-frame grounding head. It is therefore not possible to tell from the reported experiments whether the proposed grounding contributes anything beyond what the invariance loss alone would give. On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy.
Response to W1-3: We thank the reviewer for this suggestion. We have added a comparison (Table 5) with a per-frame head of comparable parameter count, retaining the same physical-state supervision, invariance objective, and other training settings. All models use Gaussian CEM without an action prior. Temporal grounding achieves higher mean success rates than the per-frame head in both datasets, supporting the effectiveness of temporal aggregation.
Table 5. Per-frame versus temporal physical grounding.
| Dataset | Grounding module | Success rate (%) |
|---|---|---|
| PushT-Ada | Per-frame | 70.53 ± 14.85 |
| PushT-Ada | Temporal grounding (Ours) | 77.60 ± 2.31 |
| OGBench Cube | Per-frame | 75.87 ± 1.03 |
| OGBench Cube | Temporal grounding (Ours) | 78.60 ± 4.12 |
“On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy.” We agree that pairing both world models with a diffusion prior and CEM removes the success-rate gap on the original Cube benchmark. To investigate this observation, we additionally evaluate Cube-Challenging at a 50-step start–goal separation with different amounts of world-model training data, while training action priors on the full dataset in every setting. As shown in Table 6, G-JEPA achieves higher mean success rates than LeWM under direct prior execution at all three world-model data proportions. However, adding CEM does not further improve performance over direct prior execution on this task set.
Table 6. Planning performance on 50-step Cube-Challenging tasks. Parentheses indicate the improvement over LeWM.
| Method | Temporal grounding | Action prior | CEM | 25% world-model data | 50% world-model data | 100% world-model data |
|---|---|---|---|---|---|---|
| LeWM | ✗ | ✗ | ✓ | 21.67 ± 3.21 | 14.67 ± 1.53 | 24.67 ± 1.53 |
| LeWM + AD w/o CEM | ✗ | ✓ | ✗ | 73.67 ± 6.11 | 85.33 ± 3.79 | 94.33 ± 3.51 |
| LeWM + AD | ✗ | ✓ | ✓ | 70.67 ± 0.58 | 78.33 ± 4.16 | 90.67 ± 2.31 |
| G-JEPA w/o AD | ✓ | ✗ | ✓ | 19.00 ± 1.73 (-2.67) | 20.33 ± 2.52 (+5.67) | 31.33 ± 2.08 (+6.67) |
| G-JEPA w/o CEM | ✓ | ✓ | ✗ | 81.33 ± 3.06 (+7.67) | 87.33 ± 2.52 (+2.00) | 95.00 ± 1.00 (+0.67) |
| G-JEPA | ✓ | ✓ | ✓ | 73.33 ± 1.15 (+2.67) | 78.33 ± 3.21 (+0.00) | 94.33 ± 1.15 (+3.67) |
We thank the reviewer for the detailed assessment and constructive suggestions. We address your concerns point by point below.
Response to W1-1: We thank the reviewer for the valuable comments. Existing JEPA-based world models have already explored physical grounding through per-frame state supervision (PhyLatent) and pairwise transition supervision (PSG-JEPA). We explore this direction to temporal physical grounding, aggregating context across frames to supervise task-relevant physical states from both encoded observations and predicted latents. Table 1 summarizes the grounding mechanisms.
Table 1. Comparison of physical grounding mechanisms.
When using these JEPA-based world models for planning, we further observed a substantial discrepancy (Table 2) between world-model predicted and actual executed outcomes, even for highly ranked action proposals. The following PushT example uses a LeWM candidate ranked among the top ten by predicted latent cost. This motivates learning an action prior conditioned on physically grounded latent representations, using learned behavioral information to guide proposal generation. The two components therefore form a progressive framework for addressing the representation learning and action planning limitations of existing JEPA-based world models.
Table 2. Predicted and executed goal errors for a highly ranked PushT action sequence.
We acknowledge that mixing policy samples with Gaussian search is also used in TD-MPC. For a clearer comparison, we have added evaluations of alternative planners and goal-conditioned TD-MPC2, as shown in Table 3. The four G-JEPA variants share the same frozen world model and matched planning budgets within each dataset. TD-MPC2 is independently trained on the same offline data with sparse goal-reaching rewards and retains its own planner. Our prior-guided planner achieves the highest mean success rate in both datasets. To examine which components contribute to the gains, we report the component and recurrence ablations in our responses to W1-2 and W1-3 below.
Table 3. Comparison with alternative planners and goal-conditioned TD-MPC2.