acceptodds
Under review as a conference paper at ICLR 2027

Exploring Temporal Context-Aware Physical Grounding in Latent World Models

Abstract

Learning latent dynamics is fundamental to building action-conditioned world models for planning. Among existing families, Joint Embedding Predictive Architectures (JEPAs) offer a natural framework by modeling transitions directly in latent space rather than at the pixel level. However, two issues remain in JEPA-based world models: (i) physical grounding lacks explicit aggregation of temporal context and (ii) action planning always overly relies on generic sampling distributions. To this end, we propose -JEPA, which incorporates temporal context into physical grounding for latent world models and couples the learned models with prior-guided action planning. Specifically, during learning, we incorporate a shared causal temporal module that aggregates temporal context within observation and prediction sequences under physical state supervision, grounding both in task-relevant physical states. When planning, we further introduce action inference through stepwise conditioning of a generative action distribution on the current and goal latent representations together with their evolving discrepancy, yielding anchored proposals that guide model-based online refinement. Together, physical state supervision guides temporal representation learning, while the action prior guides search through the learned world model. Experiments demonstrate improved planning performance on manipulation tasks, including tasks with increasing temporal separation between initial and goal observations. Code will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world model, joint‑embedding predictive architecture, physical state grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LD75OpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes G-JEPA, an end-to-end JEPA world model that augments latent prediction with physical-state supervision. A shared causal S4D module aggregates temporal context over both the observed and the predicted latent sequences and regresses the corresponding physical states. For planning, it trains a latent goal-conditioned diffusion action prior and mixes its samples with Gaussian CEM proposals under a shared world-model scoring loop. Experiments on PushT, OGBench Cube, and TwoRooms report higher planning success than the retrained baselines on the manipulation tasks, with ablations over the grounding losses, temporal architecture, proposal mixture ratio, and action-selection strategy.

Critical strengths

The paper shows a substantial and consistent improvement over the evaluated baselines on PushT, with or without the diffusion prior, and supports this with a fairly complete set of ablations over the grounding losses, the temporal architecture, the action-selection strategy, and the proposal ratio. This establishes that the system works and makes it possible to trace where the gains come from.

Critical weaknesses
  • The contribution is incremental, and the paper’s own ablations do not show that the new parts are responsible for the gains. Supervising JEPA latents with physical states was already introduced by PhyLatent and PSG-JEPA. The photometric invariance loss is a standard self-supervised objective that PhyLatent also uses. Mixing policy samples into Gaussian CEM is the planner used in TD-MPC. The only new elements are therefore the temporal S4D module and the grounding of predicted latents. However, the ablation in Table 3 adds the losses cumulatively, so the invariance loss is never evaluated on its own. On PushT, state grounding alone does not improve over LeWM, the large gain appears only when invariance is added, and grounding the predicted latents adds nothing further. The temporal module itself is never compared against a per-frame grounding head. It is therefore not possible to tell from the reported experiments whether the proposed grounding contributes anything beyond what the invariance loss alone would give. On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy. This is consequential because the method’s main results are not shown to depend on its new components.

  • The chosen tasks do not require temporal context, so the central claim cannot be tested on them. The motivation for temporal grounding is that the physical state may need to be inferred from evidence distributed across several frames. However, PushT and OGBench Cube are both rendered from a single fixed viewpoint, and the physical state is almost entirely visible in each frame. Apart from the PushT agent velocity and transient occlusion of the cube by the gripper, every grounding target can be recovered from the current image. On such tasks, a module that aggregates history has little to add, and the paper reports no setting with partial observability or ambiguous single frames where it would. Moreover, Appendix A.2 states that training uses four-frame windows, so the grounding module never aggregates more than three observed frames. The temporal context available to it is therefore very short. The paper also reports no history-length ablation and no physical-state prediction error, so there is no direct evidence that the S4D module uses temporal context at all. In addition, the module is discarded at planning time. Its effect on planning can therefore only be indirect, through the representation it shapes during training, and this effect is never measured. This is consequential because the experimental design leaves the main claim without any test that could have falsified it.

Questions for authors
  • What direct evidence shows that temporal aggregation contributes to the results? The grounding module sees at most three observed frames during training (Appendix A.2), and PushT and Cube are almost fully observable from a single frame. Can the authors report (a) success with the S4D module replaced by a per-frame grounding head of matched capacity, and (b) success as a function of training context length, for example 1, 3, 8, and 16 frames?

  • Can the authors show results on a setting where history is actually needed, for example the same tasks with partial occlusion of the block or cube at varying levels? This is where the claimed advantage should appear, and no such setting is evaluated.

  • In Table 1, PSG-JEPA performs far below the ungrounded LeWM on PushT-Ada, even though it adds physical-state supervision. Since the PushT configuration for PSG-JEPA was added by the authors, can they clarify how it was implemented, including the state targets used and whether its hyperparameters were tuned, and explain why a grounded model would fall this far below an ungrounded one?

10 comments

We thank the reviewer for the detailed assessment and constructive suggestions. We address your concerns point by point below.

W1-1: The contribution is incremental, and the paper’s own ablations do not show that the new parts are responsible for the gains. Supervising JEPA latents with physical states was already introduced by PhyLatent and PSG-JEPA. The photometric invariance loss is a standard self-supervised objective that PhyLatent also uses. Mixing policy samples into Gaussian CEM is the planner used in TD-MPC. The only new elements are therefore the temporal S4D module and the grounding of predicted latents.

Response to W1-1: We thank the reviewer for the valuable comments. Existing JEPA-based world models have already explored physical grounding through per-frame state supervision (PhyLatent) and pairwise transition supervision (PSG-JEPA). We explore this direction to temporal physical grounding, aggregating context across frames to supervise task-relevant physical states from both encoded observations and predicted latents. Table 1 summarizes the grounding mechanisms.

Table 1. Comparison of physical grounding mechanisms.

GroundingPhyLatentPSG-JEPAOurs
Encoded-latent grounding✓✓✓
Predicted-latent grounding✓✗✓
Pairwise transition grounding✗✓✗
Causal temporal grounding✗✗✓

When using these JEPA-based world models for planning, we further observed a substantial discrepancy (Table 2) between world-model predicted and actual executed outcomes, even for highly ranked action proposals. The following PushT example uses a LeWM candidate ranked among the top ten by predicted latent cost. This motivates learning an action prior conditioned on physically grounded latent representations, using learned behavioral information to guide proposal generation. The two components therefore form a progressive framework for addressing the representation learning and action planning limitations of existing JEPA-based world models.

Table 2. Predicted and executed goal errors for a highly ranked PushT action sequence.

OutcomePosition error (px)Orientation error (°)
World-model prediction1.180.095
Actual execution118.9781.93

We acknowledge that mixing policy samples with Gaussian search is also used in TD-MPC. For a clearer comparison, we have added evaluations of alternative planners and goal-conditioned TD-MPC2, as shown in Table 3. The four G-JEPA variants share the same frozen world model and matched planning budgets within each dataset. TD-MPC2 is independently trained on the same offline data with sparse goal-reaching rewards and retains its own planner. Our prior-guided planner achieves the highest mean success rate in both datasets. To examine which components contribute to the gains, we report the component and recurrence ablations in our responses to W1-2 and W1-3 below.

Table 3. Comparison with alternative planners and goal-conditioned TD-MPC2.

MethodPushT-Ada (%)OGBench Cube (%)
G-JEPA + Gaussian CEM79.43 ± 1.5878.60 ± 4.12
G-JEPA + iCEM83.46 ± 2.5170.60 ± 5.17
G-JEPA + POPLIN-A-Init84.77 ± 0.7898.00 ± 1.33
G-JEPA + prior-guided CEM (ours, ρ = 0.2)87.11 ± 1.17100.00 ± 0.00
TD-MPC27.81 ± 0.1357.53 ± 2.32

W1-2: However, the ablation in Table 3 adds the losses cumulatively, so the invariance loss is never evaluated on its own. On PushT, state grounding alone does not improve over LeWM, the large gain appears only when invariance is added, and grounding the predicted latents adds nothing further.

Response to W1-2: We thank the reviewer for pointing out the missing isolated ablation. We have added an invariance-only setting and extended the evaluation to two additional challenging task sets to examine the individual and combined effects of these components. All settings use Gaussian CEM without an action prior. As shown in Table 4, neither state supervision nor invariance alone consistently improves over LeWM. Combining them improves over either component alone on all datasets. We agree that adding predicted-state supervision does not increase mean success on PushT-Ada. It improves mean success on Cube and both challenging task sets. These results support the contribution of predicted-state supervision in several settings, with larger improvements on the challenging task sets.

Table 4. Component ablations of temporal physical grounding and nuisance invariance.

State (S)Invariance (I)Predicted state (P)PushT-Ada (%)OGBench Cube (%)PushT-Challenging(25 steps) (%)Cube-Challenging(25 steps) (%)
✗✗✗56.90 ± 2.3572.80 ± 2.5380.69 ± 1.6515.00 ± 2.65
✓✗✗55.73 ± 1.5875.40 ± 1.6576.98 ± 0.7926.00 ± 1.00
✗✓✗19.40 ± 1.2674.20 ± 1.7551.59 ± 2.3819.33 ± 5.13
✓✓✗79.43 ± 0.9076.20 ± 2.5788.89 ± 2.3826.67 ± 2.89
✓✓✓79.43 ± 1.5878.60 ± 4.1292.06 ± 2.3833.00 ± 1.73


W1-3: The temporal module itself is never compared against a per-frame grounding head. It is therefore not possible to tell from the reported experiments whether the proposed grounding contributes anything beyond what the invariance loss alone would give. On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy.

Response to W1-3: We thank the reviewer for this suggestion. We have added a comparison (Table 5) with a per-frame head of comparable parameter count, retaining the same physical-state supervision, invariance objective, and other training settings. All models use Gaussian CEM without an action prior. Temporal grounding achieves higher mean success rates than the per-frame head in both datasets, supporting the effectiveness of temporal aggregation.

Table 5. Per-frame versus temporal physical grounding.

DatasetGrounding moduleSuccess rate (%)
PushT-AdaPer-frame70.53 ± 14.85
PushT-AdaTemporal grounding (Ours)77.60 ± 2.31
OGBench CubePer-frame75.87 ± 1.03
OGBench CubeTemporal grounding (Ours)78.60 ± 4.12

“On Cube, the improvement disappears once the baseline world model is paired with the same diffusion policy.” We agree that pairing both world models with a diffusion prior and CEM removes the success-rate gap on the original Cube benchmark. To investigate this observation, we additionally evaluate Cube-Challenging at a 50-step start–goal separation with different amounts of world-model training data, while training action priors on the full dataset in every setting. As shown in Table 6, G-JEPA achieves higher mean success rates than LeWM under direct prior execution at all three world-model data proportions. However, adding CEM does not further improve performance over direct prior execution on this task set.

Table 6. Planning performance on 50-step Cube-Challenging tasks. Parentheses indicate the improvement over LeWM.

MethodTemporal groundingAction priorCEM25% world-model data50% world-model data100% world-model data
LeWM✗✗✓21.67 ± 3.2114.67 ± 1.5324.67 ± 1.53
LeWM + AD w/o CEM✗✓✗73.67 ± 6.1185.33 ± 3.7994.33 ± 3.51
LeWM + AD✗✓✓70.67 ± 0.5878.33 ± 4.1690.67 ± 2.31
G-JEPA w/o AD✓✗✓19.00 ± 1.73 (-2.67)20.33 ± 2.52 (+5.67)31.33 ± 2.08 (+6.67)
G-JEPA w/o CEM✓✓✗81.33 ± 3.06 (+7.67)87.33 ± 2.52 (+2.00)95.00 ± 1.00 (+0.67)
G-JEPA✓✓✓73.33 ± 1.15 (+2.67)78.33 ± 3.21 (+0.00)94.33 ± 1.15 (+3.67)
Reviewer_j5WxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes G-JEPA, a Joint-Embedding Predictive Architecture (JEPA) built on LeWM. The first contribution is an additional "physical grounding" pathway, in which a recurrent model (S4D) is trained to encode and decode ground-truth state information sts_t from the world model's latent state and a recurrent memory hth_t of the past. The second contribution adapts CEM-based planning by initializing the action distribution with a policy network. The authors show that G-JEPA outperforms other JEPA variants on several control tasks. They provide some evidence of where this advantage comes from by ablating the losses, the choice of recurrent architecture, and the policy-network prior.

Critical strengths
  • Clear writing. The paper is clearly written and easy to follow.

  • Evaluation on a good set of environments. The approach is evaluated on PushT-Ada, OGBench Cube, and TwoRooms, which together provide decent task coverage.

Critical weaknesses
  • Weak and disjoint contributions. Both contributions are minor modifications of a working recipe: one adds temporal information for state grounding, while the other improves CEM. The CEM improvement in particular is not novel, as initializing the action distribution with a learned network is a fairly established technique, e.g., POPLIN (Wang & Ba, 2020). Furthermore, the CEM improvement is not tied to the world-modeling approach at all. Moreover, the "physical grounding" contribution appears to be at odds with the design principles of JEPA architectures. "Physical grounding" is added by encoding and decoding a ground-truth state, which provides privileged ground-truth information during training. The main advantage of JEPA architectures over generative world models such as Dreamer (Hafner et al., 2025) has been articulated as predicting in latent space without requiring reconstruction, which allows the world model to focus on task-relevant aspects. Decoding and predicting a state seems at odds with this design philosophy and makes the approach largely dependent on how informative the provided state is. Finally, the two contributions are largely disjoint and do not form a clear, unified direction for the paper. This is the main driver of my recommendation. Both contributions are incremental, the planning improvement is not novel, and the grounding pathway departs from the principle that motivates JEPA in the first place. As a result, the paper offers little new insight.

  • Missing baselines and ablations. Several key comparisons are missing. In particular, an ablation that retains state decoding but omits the recurrence is necessary. The CEM variant should be compared against other simple CEM modifications, e.g., iCEM (Pinneri et al., 2020). Since the state information is hand-designed, it would be interesting to see what happens when it contains task-irrelevant information, which is the key disadvantage of decoding. Furthermore, it would be interesting to compare against non-JEPA baselines, e.g., TD-MPC2 (Hansen et al., 2024) adapted to the goal-conditioned setting, for instance with a sparse goal-reaching reward. Without these comparisons, it remains unclear which component drives the reported gains and how the method compares to established alternatives.

Questions for authors

Questions for Authors

  1. State decoding. State decoding seems to violate the JEPA design philosophy of being decoder-free, which is meant to keep the model from focusing on task-irrelevant details. In this setup, it appears to work because the state is designed to carry only relevant information. Could you explain why this does not contradict the JEPA principle? What would happen if the states contained task-irrelevant information? And in more realistic applications such as real-world robotic manipulation, where would the ground-truth state information (e.g., object poses) come from?
  2. Source of data. What data is the diffusion policy trained on? Does it require expert demonstrations, or would random actions or play data suffice?
  3. CEM comparisons. How does the policy-distillation approach compare to other improvements of the CEM recipe that do not require adaptations of the world model, such as those introduced in iCEM (Pinneri et al., 2020)?
  4. Recurrence ablation. The recurrence is not ablated. It would be important to see how performance is affected by providing state information as targets and reconstructing it alone, i.e., without any recurrence. PSG-JEPA does not isolate this factor either, as it appears to supervise state changes between latents. As it stands, it is unclear whether the performance gains come from the privileged state information or from the temporal information. Does the recurrence contribute to performance?
  5. Nuisance-invariant physical state. Why is this component necessary? Is it related to the physical grounding, or is it an additional contribution aimed at improving robustness? What happens if this alone is added to LeWM or just this component is removed from the loss?
  6. Physics grounding. The paper argues for physics grounding. I find this framing vague, since what is provided is not physics per se but privileged state information that is otherwise only partially observable from the image observations. Similar explicit state information could be helpful in non-physical tasks, e.g., grid-world environments such as MiniGrid, where events like key pick-ups could be provided explicitly. Could the authors describe what specifically makes this approach physics-grounded?
  7. Non-JEPA baselines. How would your method fare against non-JEPA architectures in the same goal-conditioned setting, e.g., TD-MPC2 trained with a sparse goal-reaching reward?

References

  • iCEM: Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Jörg Stückler, Michal Rolínek, and Georg Martius. Sample-efficient Cross-Entropy Method for Real-time Planning. CoRL 2020.
  • POPLIN: Tingwu Wang and Jimmy Ba. Exploring Model-based Planning with Policy Networks. ICLR 2020.
  • TD-MPC2: Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, Robust World Models for Continuous Control. ICLR 2024.
  • Dreamer 4: Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training Agents Inside of Scalable World Models. arXiv:2509.24527, 2025.

Suggestions

  • The fonts in Figure 1 are a bit small. Please enlarge them so the figure is readable without zooming in too much.
9 comments

We thank the reviewer for the careful assessment and constructive feedback. We address your concerns point by point below.

W1-1: Weak and disjoint contributions. Both contributions are minor modifications of a working recipe: one adds temporal information for state grounding, while the other improves CEM. The CEM improvement in particular is not novel, as initializing the action distribution with a learned network is a fairly established technique, e.g., POPLIN (Wang & Ba, 2020). Furthermore, the CEM improvement is not tied to the world-modeling approach at all.

Response to W1-1: We thank the reviewer for the valuable comments. Existing JEPA-based world models have already explored physical grounding through per-frame state supervision (PhyLatent) and pairwise transition supervision (PSG-JEPA). We therefore extend this direction to temporal physical grounding, aggregating context across frames to supervise task-relevant physical states.

When using these JEPA-based world models for planning, we further observed a substantial discrepancy (Table 1) between world-model predicted and executed outcomes, even for highly ranked action proposals. This motivates learning an action prior conditioned on physically grounded latent representations, using learned behavioral information to guide proposal generation. The two components therefore form a progressive framework for addressing the representation learning and action planning limitations of existing JEPA-based world models.

In addition, our method is different from POPLIN regarding the concrete planning mechanisms. POPLIN searches over perturbations to policy-generated actions or policy parameters, whereas our method combines samples from a learned action prior and an evolving Gaussian search distribution. For a clearer comparison, we have added evaluations of different planning protocols, including iCEM and POPLIN, with the results presented in our response to Q3 below.

Table 1. Predicted and executed goal errors for a highly ranked PushT action sequence by using LeWM.

OutcomePosition error (px)Orientation error (°)
World-model prediction1.180.095
Actual execution118.9781.93

W1-2: Moreover, the "physical grounding" contribution appears to be at odds with the design principles of JEPA architectures. "Physical grounding" is added by encoding and decoding a ground-truth state, which provides privileged ground-truth information during training. The main advantage of JEPA architectures over generative world models such as Dreamer (Hafner et al., 2025) has been articulated as predicting in latent space without requiring reconstruction, which allows the world model to focus on task-relevant aspects. Decoding and predicting a state seems at odds with this design philosophy and makes the approach largely dependent on how informative the provided state is. Finally, the two contributions are largely disjoint and do not form a clear, unified direction for the paper. This is the main driver of my recommendation. Both contributions are incremental, the planning improvement is not novel, and the grounding pathway departs from the principle that motivates JEPA in the first place. As a result, the paper offers little new insight.

Q1: State decoding. State decoding seems to violate the JEPA design philosophy of being decoder-free, which is meant to keep the model from focusing on task-irrelevant details. In this setup, it appears to work because the state is designed to carry only relevant information. Could you explain why this does not contradict the JEPA principle? What would happen if the states contained task-irrelevant information? And in more realistic applications such as real-world robotic manipulation, where would the ground-truth state information (e.g., object poses) come from?

Response to W1-2 and Q1: We thank the reviewer for the valuable comments and appreciate the opportunity to clarify this motivation. At first, physical states refer to task-relevant physical quantities, such as object position, orientation, velocity, and robot joint angles. These quantities serve as supervision targets during training rather than inputs to the image encoder. However, existing JEPA methods encode objects and their visual context into latent representations without explicitly constraining which physical information should be preserved, inspiring PhyLatent and PSG-JEPA for per-frame state and pairwise transition supervision, respectively. Furthermore, we extend this direction to temporal physical grounding, aggregating context across frames to supervise task-relevant physical quantities from both encoded observations and predicted latents. These supervision constraints on the information retained are similar to the design in which LeWM introduces Gaussian regularization to constrain the latent distribution. Therefore, both our design and previous physics-based methods (PhyLatent & PSG-JEPA) do not contradict JEPA’s philosophy.

Regarding the source of supervision, our experimental state labels are obtained directly from the simulator. In real-world manipulation, robot joint measurements can be obtained from proprioceptive sensors, while object poses could be estimated from other tools, such as calibrated RGB-D observations or pretrained depth foundation models.


Q2: Source of data. What data is the diffusion policy trained on? Does it require expert demonstrations, or would random actions or play data suffice?

Response to Q2: We train the diffusion policy on expert demonstrations, using the same offline trajectories as world-model training without collecting additional demonstrations. We will clarify more details about the data source in the Section 3.3 Action Prior Acquisition.


W2-1: Missing baselines and ablations. Several key comparisons are missing. In particular, an ablation that retains state decoding but omits the recurrence is necessary. The CEM variant should be compared against other simple CEM modifications, e.g., iCEM (Pinneri et al., 2020). Since the state information is hand-designed, it would be interesting to see what happens when it contains task-irrelevant information, which is the key disadvantage of decoding.

Q3: CEM comparisons. How does the policy-distillation approach compare to other improvements of the CEM recipe that do not require adaptations of the world model, such as those introduced in iCEM (Pinneri et al., 2020)?

Response to W2-1 and Q3: We appreciate these valuable suggestions. We added a comparison with other alternative planners (baseline, iCEM, and POPLIN-A-Init) in Table 2. Our action prior-guided planner achieves the highest mean success rate in both datasets.

Table 2. Comparison with alternative planners.

MethodPushT-Ada (%)OGBench Cube (%)
Gaussian CEM79.43 ± 1.5878.60 ± 4.12
iCEM83.46 ± 2.5170.60 ± 5.17
POPLIN-A-Init¹84.77 ± 0.7898.00 ± 1.33
Ours87.11 ± 1.17100.00 ± 0.00

¹ POPLIN-A-Init is a variant of POPLIN.


W2-2:Furthermore, it would be interesting to compare against non-JEPA baselines, e.g., TD-MPC2 (Hansen et al., 2024) adapted to the goal-conditioned setting, for instance with a sparse goal-reaching reward. Without these comparisons, it remains unclear which component drives the reported gains and how the method compares to established alternatives.

Q7: Non-JEPA baselines. How would your method fare against non-JEPA architectures in the same goal-conditioned setting, e.g., TD-MPC2 trained with a sparse goal-reaching reward?

Response to W2-2 and Q7: We thank the reviewer for this fair suggestion. We have added a comparison with goal-conditioned TD-MPC2, as shown in Table 3. G-JEPA achieves higher success rates on both tasks under this offline goal-conditioned setting. On PushT-Ada, trajectory inspection shows that TD-MPC2 selects actions that deviate substantially from the training action distribution, which may contribute to its low success rate. Table 3. Comparison with None-JEPA architecture.

MethodPushT-Ada (%)OGBench Cube (%)
Ours87.11 ± 1.17100.00 ± 0.00
TD-MPC2²7.81 ± 0.1357.53 ± 2.32

² TD-MPC2 is trained on the same offline data with sparse goal-reaching rewards and uses its own planner.


Q4: Recurrence ablation. The recurrence is not ablated. It would be important to see how performance is affected by providing state information as targets and reconstructing it alone, i.e., without any recurrence. PSG-JEPA does not isolate this factor either, as it appears to supervise state changes between latents. As it stands, it is unclear whether the performance gains come from the privileged state information or from the temporal information. Does the recurrence contribute to performance?

Response to Q4: We thank the reviewer for this suggestion. To isolate the contribution of recurrence, we replace the temporal grounding module with a per-frame head of comparable parameter count, while retaining the same physical-state supervision and other training settings. And both variants use Gaussian CEM without an action prior. As shown in Table 4, temporal physical grounding achieves higher mean success rates than the per-frame head on both tasks under the same physical-state supervision. These results support the contribution of temporal aggregation, showing that the observed gains are not explained by privileged state supervision alone.

Table 4. Per-frame versus temporal physical grounding.

DatasetGrounding moduleSuccess rate (%)
PushT-AdaPer-frame70.53 ± 14.85
PushT-AdaTemporal physical grounding (Ours)77.60 ± 2.31
OGBench CubePer-frame75.87 ± 1.03
OGBench CubeTemporal physical grounding (Ours)78.60 ± 4.12
Reviewer_vhm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this work introduces G-JEPA, an action-conditioned latent world model, which enhances robotic planning by combining temporal physical grounding with prior-guided action selection. It leverages a shared causal S4D module to track physical states across both observed and predicted sequences, paired with a hybrid planner that integrates a diffusion action prior with online Gaussian CEM refinement. Experiments show that G-JEPA outperforms existing models like LeWM, TD-JEPA, and PSG-JEPA on manipulation tasks, particularly across extended time horizons.

Critical strengths
  1. temporal physical grounding is novel. Unlike static frame-by-frame or isolated latent-pair grounding (e.g. PhyLatent, PSG-JEPA), the paper integrates S4D layers to accumulate multi-step temporal context, whcih guides the encoder and predictor towards task-relevant physical dynamics.
  2. hybrid action planning. the integration of an offline-learned diffusion action prior with online CEM search, via a shared elite pool, bridges the gap between behavioral cloning priors and model-based refinement.
  3. extensive empirical validation.
Critical weaknesses
  1. confounding long-horizon results. for K>5K > 5 in PushT-Distance, the diffusion prior supplies the first five action chunks, this makes it unclear if 75-step gains come from the grounded world model or strong open-loop diffusion trajectories.
  2. missing latency analysis. combining DDIM sampling steps with online CEM rollouts can introduce heavy compute overhead, but the paper provides no runtime or latency comparisons against baselines.
  3. saturated baselines. tasks like OGBench Cube and TwoRooms reach near 100% success with direct prior execution or ungrounded variants, which reduces G-JEPA's actual impact in simpler settings.
Questions for authors

1.would it be sensible to provide an ablation for PushT-Distance (K>5K > 5) with the diffusion prior disabled to isolate the world model's contribution from the initial open-loop action chunks? 2. what is the wall-clock execution time per replan step given the combined DDIM and CEM overhead, and how does it compare to LeWM or a pure Diffusion Policy?

6 comments

We thank the reviewer for the constructive feedback and for recognizing the value of temporal physical grounding, prior-guided planning, and our empirical evaluation. We address your concerns point by point below.

W1: Confounding long-horizon results. For K > 5 in PushT-Distance, the diffusion prior supplies the first five action chunks. This makes it unclear if 75-step gains come from the grounded world model or strong open-loop diffusion trajectories.

Q1: Would it be sensible to provide an ablation for PushT-Distance (K > 5) with the diffusion prior disabled to isolate the world model's contribution from the initial open-loop action chunks?

Response to W1 and Q1: We thank the reviewer for this suggestion. We reorganized the 75-step (K>5K>5) results on PushT-Distance and added experiments on PushT-Challenging as in Table 1. The requested ablation without the diffusion prior corresponds with the result in the Table 2 of paper, denoted as “G-JEPA w/o AD.” The reorganized results are summarized below. As shown in Table 1, G-JEPA world model improves planning performance over the LeWM baseline. Adding the diffusion prior further improves performance with both world models.

Table 1. World-model and action-prior ablations on 75-step PushT tasks.

MethodTemporal groundingAction Diffusion priorCEMPushT-Distance (%)PushT-Challenging (%)
LeWM✗✗✓16.54 ± 0.6010.97 ± 0.64
LeWM + AD✗✓✓23.44 ± 1.4118.33 ± 1.50
G-JEPA w/o AD✓✗✓47.53 ± 1.6344.03 ± 1.73
G-JEPA✓✓✓55.21 ± 0.9052.22 ± 0.96

W2: Missing latency analysis. Combining DDIM sampling steps with online CEM rollouts can introduce heavy compute overhead, but the paper provides no runtime or latency comparisons against baselines.

Q2: What is the wall-clock execution time per replan step given the combined DDIM and CEM overhead, and how does it compare to LeWM or a pure Diffusion Policy?

Response to W2 and Q2: We thank the reviewer for this practically relevant suggestion. We have added wall-clock latency measurements per replanning step for variants with and without DDIM sampling and CEM, as summarized in Table 2. All methods were measured on the same exclusively used H100 GPU, with 300 timed replanning calls per method and dataset after warm-up. Compared with LeWM, G-JEPA incurs additional latency from DDIM sampling, but its average replanning latency remains within the same order of magnitude and below 0.5 seconds on both tasks. These results indicate that the additional planning overhead is manageable under our evaluated settings.

Table 2. Wall-clock latency per replanning step.

MethodTemporal groundingAction Diffusion prior(DDIM)CEMPushT-Ada (s/replan)OGBench Cube (s/replan)
LeWM✗✗✓0.312 ± 0.0940.295 ± 0.093
G-JEPA w/o AD✓✗✓0.285 ± 0.0890.296 ± 0.075
G-JEPA w/o CEM✓✓✗0.155 ± 0.0660.036 ± 0.031
G-JEPA✓✓✓0.450 ± 0.1110.316 ± 0.070

W3: Saturated baselines. Tasks like OGBench Cube and TwoRooms reach near 100% success with direct prior execution or ungrounded variants, which reduces G-JEPA's actual impact in simpler settings.

Response to W3: We thank the reviewer for highlighting this limitation. We conduct additional evaluations on two task sets: 100 Cube-Challenging tasks (Table 3a) with larger object displacements and 50-step start–goal separations, and 50 Scene tasks (Table 3b). Additionally, to avoid saturated baselines, we further evaluate methods across various training data proportions. As shown in Table 3a, G-JEPA achieves higher mean success rates than LeWM under direct prior execution at all three world-model data proportions. The action prior trained on the full dataset already provides effective actions on Cube-Challenging, and adding CEM does not further improve performance. In contrast, Table 3b shows that G-JEPA benefits from combining learned proposals with CEM on Scene, outperforming both its direct-prior and Gaussian-CEM variants at all three data proportions. Together, these results support the effectiveness of G-JEPA relative to the corresponding LeWM baselines.

Table 3a. Planning performance on 50-step Cube-Challenging tasks. Parentheses indicate the improvement over LeWM.

MethodTemporal groundingAction priorCEM25% world-model data50% world-model data100% world-model data
LeWM✗✗✓21.67 ± 3.2114.67 ± 1.5324.67 ± 1.53
LeWM + AD w/o CEM✗✓✗73.67 ± 6.1185.33 ± 3.7994.33 ± 3.51
LeWM + AD✗✓✓70.67 ± 0.5878.33 ± 4.1690.67 ± 2.31
G-JEPA w/o AD✓✗✓19.00 ± 1.73 (-2.67)20.33 ± 2.52 (+5.67)31.33 ± 2.08 (+6.67)
G-JEPA w/o CEM✓✓✗81.33 ± 3.06 (+7.67)87.33 ± 2.52 (+2.00)95.00 ± 1.00 (+0.67)
G-JEPA✓✓✓73.33 ± 1.15 (+2.67)78.33 ± 3.21 (+0.00)94.33 ± 1.15 (+3.67)

Table 3b. Planning performance on Scene tasks. Parentheses indicate the improvement over LeWM.

MethodTemporal groundingAction priorCEM25% world-model data50% world-model data100% world-model data
LeWM✗✗✓38.00 ± 5.2924.00 ± 2.0033.33 ± 5.03
LeWM + AD w/o CEM✗✓✗14.67 ± 4.628.00 ± 2.0014.00 ± 2.00
LeWM + AD✗✓✓29.33 ± 2.3124.67 ± 2.3128.00 ± 0.00
G-JEPA w/o AD✓✗✓28.00 ± 3.46 (-10.00)20.00 ± 2.00 (-4.00)26.00 ± 4.00 (-7.33)
G-JEPA w/o CEM✓✓✗32.67 ± 6.11 (+18.00)40.00 ± 5.29 (+32.00)36.00 ± 2.00 (+22.00)
G-JEPA✓✓✓68.67 ± 5.03 (+39.33)74.67 ± 2.31 (+50.00)70.67 ± 1.15 (+42.67)

Thank you for the detailed responses and additional experiments, especially the new PushT ablations, runtime comparisons, and evaluations on more challenging Cube and Scene tasks, which addressed several of my original concerns.

The main strengths remain the use of temporal information to improve physical state representations and the combination of learned action proposals with CEM planning, while the main limitations are the reliance on ground-truth physical states during training, the extra computational cost, and the varying benefits across tasks.

Although the new results provide stronger support for the proposed approach, I still have some concerns about how well it generalizes, and how much each component contributes to the overall improvement, so I appreciate the authors' efforts but will keep my original weak-reject rating.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline approaches, we would be grateful if you could take a look at our rebuttal and the additional experiments addressing your questions. We have also revised the manuscript accordingly. We would appreciate your feedback on whether these additions address your concerns and would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Best regards,

Authors of submission 8506

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.