acceptodds
Withdrawn from ICLR 2027

EquiJEPA: Learning Action Representations for Effect Prediction and Action Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An action representation should describe how an action changes the world and help an agent choose actions that produce a desired change. We introduce EquiJEPA, an inverse-dynamics Joint Embedding Predictive Architecture that learns this bidirectional connection in latent space. A state-blind effect head predicts latent changes from actions alone, encouraging transitions induced by the same action to concentrate around a shared prototype. Each transition consists of this prototype and a residual capturing state-dependent departures, including contact and collision. The prototypes provide concrete operations in latent space: an effect can be transferred to a new state, combined with another effect, and decoded into an action through the inverse head. Across simulated environments, we evaluate these operations through physical execution, including cross-state reuse and action combinations absent from training. EquiJEPA connects action-conditioned prediction with effect-conditioned action generation, providing an explicit action representation for reuse and compositional generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, World Model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o4ZfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EquiJEPA, a JEPA-based world model that encourages consistent latent displacements for the same action across difference states. The authors argue that the resulting transition concentration facilitates action prototype reuse, composition, and the detection of state dependent interactions.

Critical strengths

The paper presents a simple and intuitive approach to encouraging action equivariance in latent representations. The resulting representations exhibit interesting properties, including the composition and reuse of action prototypes, which could be useful in certain settings.

Critical weaknesses
  1. My primary concern is that the action equivariance may impose a rather restrictive assumption that does not generally hold in realistic environments. Specifically, the addition of latent displacements is commutative, whereas the transitions by the action sequences in many robotic environments are not. For example, consider two actions for a 2D dot moving toy environment: (1) rotating 90 degrees to the left and (2) moving forward one step. Performing (1) after (2) and (2) after (1) leads to different xy positions, while the action equivariance requires the latent representation of the reaching states to be identical; this can creates a fundamental conflict with the need to distinguish the resulting states. I understand that the proposed method encourages equivariance through a soft constraint rather than enforcing it exactly. Nevertheless, this example suggests a potential tradeoff between the proposed equivariance and the ability to represent general MDP. I believe the paper would benefit from a more thorough discussion of this limitation and the conditions under which the proposed assumption is reasonable.
  2. While the authors demonstrate potential applications such as action prototype reuse and the detection of state dependent effects, it is not entirely clear why these properties are particularly important for JEPA based world modeling. In addition, direct action composition is primarily evaluated in a simplified two dot environment where action effects are inherently additive. Although prototype reuse is also evaluated in Push-T, it would be helpful to further investigate whether the compositional properties extend to more realistic environments with noncommutative dynamics.
  3. The proposed method builds upon an existing inverse-dynamics JEPA [1] by introducing a relatively simple equivariance loss and an effect prediction head. While simplicity is certainly a desirable property, I am not fully convinced that the technical novelty and demonstrated practical benefits are sufficiently substantial for the top tier conference, particularly given the restrictive nature of the proposed inductive bias and the inconsistent improvements in downstream planning.

[1] Petr Ivashkov et al., Sensorimotor World Models: Perception for Action via Inverse Dynamics, arxiv, 2026.

Questions for authors
  1. Could the authors further clarify the fundamental motivation for encouraging action equivariance in world models, particularly when this property may conflict with the underlying environment dynamics? Furthermore, while it is understandable that action decodability does not necessarily imply transition concentration, the importance of concentration itself for world modeling remains unclear to me. I believe a more explicit discussion of why these properties are desirable would strengthen the paper.
  2. Have the authors considered evaluating the method on other environments involving multiple interacting objects and more complex action dynamics, e.g. OGBench Scene? Such experiments could help clarify the applicability of the proposed approach beyond environments with relatively simple action structures.
  3. The notation for expectations appears somewhat inconsistent, with square brackets used in some places but omitted in others. It would be helpful to make the notation consistent. Additionally, the first paragraph of Section 3.3 could provide a more intuitive introduction to controlled replay η2\eta^2 and held out action fit R2R^2, as these metrics are introduced rather abruptly.
  4. I wonder whether Figure 1 is the most informative result to highlight immediately after the abstract. Since EquiJEPA explicitly optimizes for transition concentration, the improvements in Figure 1 and Table 1 are somewhat expected and do not necessarily demonstrate practical utility. I believe a figure that jointly illustrates improved transition concentration and the preservation of state information, as evaluated in Tables 1 and 2, could provide a more comprehensive overview of the main findings.
Reviewer_kHUROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latent world model encodes images into vectors and predicts the next vector from the current one and an action. EquiJEPA adds one extra head that must predict the change in the latent from the action alone, without seeing the state. Because this head cannot see the state, the encoder is pushed to give the same action a similar latent change everywhere. It measures consistency with a replay test and shows large gains over the matched base model in a toy dot world, Push-T, OGBench Cube and a maze, while probes for agent and object state stay roughly unchanged. The prototypes are then used to compose actions, act from new states, detect walls, build a sparse dictionary of motions, switch command interfaces, and warm-start a planner.

Critical strengths
  • The paper identifies a real property that had not been measured before: a model can decode actions perfectly while the same action still produces very different latent changes in different states. It gives a clean test for this, and the proposed loss closes much of the gap in all four environments without hurting state information.

  • The controls are careful. A linear re-mapping of the base latents cannot reproduce the effect, the base gets a capacity-matched prototype fitted after training, planner temperatures are calibrated per method with a temperature-free check, and the planning gains are honestly traced to the warm start rather than to the dictionary.

  • The downstream demonstrations are interesting. The residual locates maze walls (AUC 0.94 against 0.66 for the base), a sparse dictionary recovers the true primitives exactly in the toy world, and a frozen dictionary still works after the command format is changed.

Critical weaknesses
  • The method is a single extra loss, and it is close to earlier homomorphic and equivariant latent models such as the homomorphism autoencoder. Proposition 1 is a one-line construction rather than theoretical support. The paper's value lies in the measurement and the careful experiments, not in a new model.

  • On Push-T, the most standard task, the base model plans better than EquiJEPA at every budget and under every solver variant. This appears only in the appendix, and the paper's own analysis ends without an explanation.

  • There is no comparison with any published world-model planner at its published budget. DINO-WM is reduced to frozen features and wrongly described as having no predictor, and LeWM is run only at the paper's own small budgets, which the paper admits are not comparable.

Questions for authors

Can you explain or resolve why the base plans better on Push-T under every solver? Also would like to see adding DINO-WM with its trained predictor and LeWM at their published budgets.

Reviewer_QDGPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigate the problem of world modelling, using a JEPA approach. Specifically, the paper adds an action-consistency loss to SMWM, a JEPA-style latent world model trained with inverse dynamics. A small network rho(a) sees only the action and must predict the change betwee the latent state z_{t+1} and z_t. This pushes each action to have the same effect in latent space, for different starting state z_t.

The method is evaluated against the SMWM base on four small simulated tasks: a dot world, Push-T, OGBench Cube and a point maze. The main result is that each action’s latent effect becomes more consistent across states on all four tasks, while probes for agent and object state stay close to the base. On Push-T, for example, the share of the latent change explained by the action alone increases when adding the new consistency loss. The authors also show that the learned action vectors can be added together and reused in new states using the a two-dot world environment and PushT.

Critical strengths
  • The idea is simple and clear, and cheap to train.
  • Consistency is measured well, by replaying the same action from many states.
  • The ablations are careful, and failures are reported.
Critical weaknesses
  • My main concern with the paper is its limited empirical validation. Paper only reports results on small-scale setup: four simple simulated tasks, a tiny encoder (ViT-Tiny). There is no real-robot data, no larger or more varied environment dataset. Additionally, in most of the tested tasks, an action is a fixed displacement of the agent in world coordinates, so it already moves the agent in almost the same way in every state; the method is therefore mostly tested where its core assumption holds by construction, not where an action’s effect truly depends on the state. It's unclear if the proposed EquiJEPA will generalize to more realistic setting.
  • There is no planning results in the main paper. Planning is the main use of a world model, but the main paper do not show any planning results. While the appendix, show some planning experiments, it only focus on small search budget (K < 32), hence we can't compare the result directly with the published results of LeWM or SMWM.
  • It's unclear if the approach helps to get better plan as on PushT, EquiJEPA leads to a degradation of performance.
Questions for authors
  • Can you put planning in the main text, with larger budgets, more seeds and a comparison to published LeWM and SMWM numbers? Why is Push-T worse?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.