acceptodds
Under review as a conference paper at ICLR 2027

AGR-JEPA: Action-Grounded Residual Joint-Embedding Predictive Architecture for Planning

Abstract

In this research, we propose Action-Grounded Residual Joint-Embedding Predictive Architecture (AGR-JEPA), a reconstruction-free latent world model for planning. Our work is based on a key observation: existing approaches typically trained to minimize next-embedding prediction error, whereas effective planning additionally requires the consequences of different actions to remain distinguishable in latent space. Through controlled experiments, we find that lower prediction error does not consistently correspond to higher planning success, indicating that latent predictability alone is not a reliable proxy for planning utility. We refer to this mismatch as the prediction–planning gap. To address this gap, AGR-JEPA adopts a residual parameterization that represents the future latent state as the combination of the current latent state and an action-conditioned latent update, focusing dynamics learning on action-induced state changes. Furthermore, AGR-JEPA introduces a control-grounding objective that uses action recoverability as a training signal, encouraging latent updates to retain information relevant to their conditioning actions and thereby improving the behavioral discriminability of action outcomes. This auxiliary objective is used only during training and incurs no additional computation at planning time. Across multiple 2D and 3D control tasks, AGR-JEPA achieves an average planning success rate of 92.06%, delivering consistent improvements over leading baselines across multiple tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA, latent world model, Control Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KC1EOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the problem of using predictive JEPA architectures for planning. Like others, the authors note that a lower prediction error does not necessarily correlate with better planning capabilities. To remedy this problem, they propose using a residual prediction scheme (predict the difference Delta_{z_t} between the next step z_{t+1} and the current one z_t) instead of the usual one (predict the next step), plus a regularizer encouraging Delta_{z_t} (in both its ground-truth and predicted forms) to predict the command that has been used to transition from z_t to z_{t+1}. This is a reasonable approach, and it improves planning performance on several benchmarks compared to other comparable methods, but the only real novelty appears to be the residual prediction, which is not well justified and dot not improve performance on its own.

Critical strengths

The method is reasonable and the combination of the residual prediction with the control-grounding regularizer gives good results.

Critical weaknesses

The main weakness is the lack of novelty of the paper. As far as I understand the "control-grounding" regularizer is a simple variant of the inverse dynamics regularizer used in Sobal et al. and ZHang et al. for example. The only true novelty I could find was in using a residual prediction of the next step, but this by itself does not improve planning performance, as noted by the authors themselves.

Questions for authors

Try to better justify the residual prediction.

The prediction-planning gap is expected, and I would like the authors to elaborate on the purpose of the corresponding section and of the counterfactual consequence discrimination section and the conclusions to be drawn from it since they were not obvious to me.

Reviewer_v8UROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper provides new loss functions to train world models. Specifically, the authors suggest that training the model to predict updates of latent state and regularizing this using action predictors helps the model improve its performance when compared to methods that only train with next latent-state prediction objectives. The authors provide sufficient empirical validation for these claims.

Critical strengths

The paper is clearly written and is easy to understand and follow. The motivations of the authors are clear and the approach they consider is novel and seemingly effective. The idea of additionally regularizing using action-differentiability is a nice way to enforce that the latent space separates candidate actions and seems reasonable to me. I appreciate the completeness of the experiment section, and the numerical results are quite promising.

Critical weaknesses

Firstly, I believe that Section 4.4 comes too late to be used as motivation for AGR-JEPA. I would have liked to see it much earlier, possibly even as early as the introduction. That being said, I also question as to why the authors did not place AGR-JEPA in this diagram. One would hope that it falls closer to the "Ideal Region" that the authors mark in Figure 4.

My major concern is regarding the lack of intuition for the need of residual loss function. If I am not mistaken, equation 3 can be rewritten as $ || \widehat{z_{t+1}} - z_{t+1} || $ which is the standard next latent state prediction loss. Given this, I am very confused by why the authors present this reformulation as a novelty since the two loss functions are equivalent. I am even more confused by why the success rate ablations using only this reformulation in Tables 2,3 show degradation in performance compared to the baseline. I would expect them to exactly match this baseline. This further begs the question as to why the authors are reporting a differentiation in performance between the standard LeWM and Residual-LeWM in Section 4.4.

Questions for authors

Given the weaknesses I list above, I would like for the authors to provide some serious justifications as to why residuals lead to improved performance and why their observed empirical improvements with residuals are not simply statistical perturbations. If not, the action-prediction loss is the only real novelty over standard LeWM and the paper needs a solid rewrite and I would not be able to recommend acceptance.

Reviewer_WfRqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reconstruction-free latent world models for goal-conditioned visual planning and argues that low next-latent prediction error alone does not yield useful action ranking. It proposes AGR-JEPA, which predicts residual latent updates and trains a training-only decoder to recover the executed action from both the target and the predicted update. The method is evaluated on four continuous-control tasks, with ablations on Push-T and a counterfactual consequence-identification diagnostic.

Critical strengths

The paper reports promising results: AGR-JEPA achieves the highest average planning success among the compared methods across four continuous-control tasks, and the proposed counterfactual consequence-identification diagnostic offers a planning-relevant way to assess latent world models beyond next-latent prediction error.

Critical weaknesses
  • Recovering actions from latent transitions as a training signal is inverse dynamics, an idea that goes back about ten years in representation learning. Previous JEPA-style world models such as PLDM and Delta-JEPA have already used this idea. What AGR-JEPA adds on top is a residual parameterization of the predictor and an extra action-recovery loss on the predicted displacement. Neither is a substantially novel, and the paper does not demonstrate a distinct advance over these past works.

  • The ablations are run only on Push-T. There, the residual parameterization alone shows reduction in success below the LeWM baseline, and adding action recovery on the predicted displacement improves only marginally over target-only grounding, within the reported variance. The component that seems to be the most effective is the action recovery on the target displacement but that seems to have an overlap with Delta-JEPA. While there is gain showed over Delta-JEPA in the OGB-Cube context, the ablation is not reported and hence it is difficult to know the reason of gained performance. Likewise, the counterfactual analysis is only presented and compared against LeWM. It is not clear why it is not compared against Delta-JEPA.

Questions for authors
  • Can the authors provide more ablations on the environments where the main gains are observed
  • The residual loss in Eq. 3 is algebraically identical to the absolute next-latent loss with an identity skip connection, as Appendix D acknowledges, yet Table 2 shows the two variants behave differently; what causes this difference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.