acceptodds
Under review as a conference paper at ICLR 2027

No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models

Abstract

Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting future embeddings, but the objective admits a trivial solution of a constant encoder, so every practical system adds an anti-collapse mechanism. LeWorldModel (LeWM) prevents collapse with SIGReg, a regularizer that forces the latent distribution to match an isotropic Gaussian: the representation is stabilized by prescribing what it must look like, independently of the environment it models. We argue that the anti-collapse pressure can instead come from the transition data itself. Action-Contrastive Masked Transition Modeling (AC-MTM) keeps LeWM's forward latent prediction objective and adds a training-only inverse-dynamics head trained with Action-NCE: each latent transition must identify the action that produced it among the other actions in the batch, a discrimination task that a collapsed encoder provably fails. The inverse branch is discarded after training, leaving test-time encoding, forward prediction, planning, and compute identical to LeWM. On four standard pixel-control tasks under a matched planning protocol, AC-MTM trains stably from scratch and matches or exceeds SIGReg. On the harder multi-object OGBench Visual Scene task, results suggest that SIGReg's prescribed geometry is a bottleneck: AC-MTM reaches success rate versus for SIGReg, improving by 20–24 points in each training seed. A random-policy run gives a 52% baseline estimate. Contrastive inverse dynamics thus provides a distribution-free anti-collapse signal that requires no target network, stop-gradient, pretrained encoder, or reconstruction objective, and we characterize the action space and observability assumptions under which it holds. We will release code and configuration files after review.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
joint-embedding predictive architectures, self-supervised representation learning, world models, contrastive learning, representation collapse, inverse dynamics, latent dynamics, model-based planning, visual control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zE6cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this work proposes AC-MTM, a distribution-free anti-collapse mechanism. The authors introduce a training-only inverse-dynamics head trained via an InfoNCE-style loss (Action-NCE) over in-batch action candidates. a theoretical lower bound (log⁡N\log N) is provided to show that a collapsed constant encoder inevitably fails this action-identification task at chance level. Large empirical evaluations across standard continuous control suites and a harder multi-object OGBench Visual Scene task demonstrate that AC-MTM trains stably without collapse, it matches SIGReg on standard benchmarks, and significantly outperforms it on complex multi-object scenes.

Critical strengths
  1. an anti-collapse mechanism without distributional prior. AC-MTM replaces heuristic global priors, e.g. isotropic Gaussian matching, with a self-supervised, data-driven pressure, derived directly from sensorimotor transitions.
  2. theoretical justification. The information-theoretic argument, which shows that a collapsed constant encoder reduces the inverse classification problem to random guessing (≥log⁡N\ge \log N), is solid.
  3. computational efficiency. AC-MTM preserves the same inference-time architecture, memory footprint, and CEM planning protocol as LeWM without extra cost.
Critical weaknesses
  1. hyperparameter dependency. The InfoNCE objective introduces sensitivity to batch composition, negative sample size, temperature τ\tau, and loss weight λ\lambda. Although a single fixed λ=0.30\lambda = 0.30 worked across the tested suite, scaling stability to diverse environments or larger batch sizes remains unverified.
  2. vulnerability in weakly-controlled state dimensions (PushT & Puzzles): as the contrastive inverse loss only penalizes representations that fail to distinguish actions, state variables with weak immediate visual sensitivity, e.g. the T-block orientation in PushT or button states in visual puzzles, are underencoded, this can lead to worse performance compared to global variance priors like SIGReg.
  3. long-horizon generalization. The gains on OGBench Visual Scene is impressive, but they rely on a customized short-horizon trajectory-goal MPC protocol (rather than the official 750-step fixed-goal leaderboard protocol, under which both methods completely fail 0/2500/250 success rate), this raises questions about long-horizon generalization.
Questions for authors
  1. as Action-NCE prioritizes features that differentiate actions, how can AC-MTM be extended or modified to ensure complete coverage of task-critical static or weakly-controlled state factors such as object orientations or hidden object states without re-introducing global distribution regularizers like SIGReg?
  2. AC-MTM and SIGReg both fail completely (0/2500/250) under the official 750-step fixed-goal OGBench Visual Scene protocol. Can the authors clarify what specific architectural or optimization bottlenecks prevent these one-stage pixel JEPAs from scaling to longer-horizon tasks, and how the short-horizon trajectory-goal protocol avoids these limitations?
  3. this method relies on in-batch action negatives to save compute. how sensitive is AC-MTM to variations in batch size BB?
Reviewer_gjWtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Action-Contrastive Masked Transition Modeling (AC-MTM), a training-only contrastive inverse-dynamics head for JEPA world models. It replaces LeWM's SIGReg Gaussian latent regularizer with Action-NCE: given a latent transition and a batch of candidate actions, the model identifies the action that produced the transition. The inverse head is removed at test time, so the planner and deployment computation remain unchanged. Experiments compare AC-MTM with SIGReg, no regularizer, and inverse-MSE across TwoRoom, Reacher, PushT, OGBench-Cube, and OGBench Visual Scene. Across three seeds, AC-MTM improves several tasks and reaches 80.0±2.0% on the authors' trajectory-goal OGBench-Scene protocol versus 58.0±2.0% for SIGReg, while underperforming SIGReg on PushT.

Critical strengths
  1. The paper presents a simple and well-motivated alternative to Gaussian anti-collapse regularization. Action-NCE directly uses action-conditioned transitions, and the inverse head is removed at inference, leaving the planner and deployment computation unchanged.

  2. The evaluation includes useful mechanism-oriented controls across diverse environments. NoReg demonstrates collapse, MTM-MSE separates inverse-dynamics supervision from the contrastive formulation, and the Reacher seed analysis evaluates stability. The PushT degradation also provides a clear and informative limitation of the proposed approach.

Critical weaknesses
  1. The Action-NCE objective relies on an identifiability assumption that is nontrivial in continuous control: the observed action must be distinguishable from the other in-batch actions given the latent transition. This can fail when different actions produce nearly identical effects because of actuator redundancy, saturation, contact dynamics, or no-op regions, and when the same action produces different effects in different states. Treating every other in-batch action as a negative may therefore penalize behaviorally equivalent actions or encourage action-distribution and trajectory-specific shortcuts rather than a robust inverse-dynamics representation. The paper mentions false negatives but does not quantify their prevalence or test whether the gains persist under an action-equivalence-aware or multi-positive contrastive objective. This limitation is central to the claimed generality of Action-NCE as a replacement for SIGReg.

  2. The auxiliary objective supervises one-step action identification, whereas the claimed downstream benefit is improved multi-step latent planning with repeated model rollouts and CEM. The paper does not establish that Action-NCE improves the multi-step prediction accuracy, error accumulation, or uncertainty properties that determine action-sequence selection. This gap is especially important because the headline Scene improvement is measured under the short-horizon trajectory-goal protocol, while both AC-MTM and SIGReg obtain 0/250 under the official long-horizon fixed-goal protocol. The current evidence therefore does not clarify whether Action-NCE improves long-horizon planning or mainly improves local one-step representation learning under the selected evaluation regime.

Questions for authors
  1. How frequently do in-batch negatives correspond to actions that are behaviorally equivalent or produce indistinguishable latent transitions? Can the authors evaluate an action-equivalence-aware or multi-positive version of Action-NCE, and report whether the advantage over SIGReg remains?

  2. How sensitive is Action-NCE to the behavior-policy action distribution used to form in-batch candidates? In particular, does the learned representation and planning performance remain stable when training data are collected by policies with different action marginals, or when candidate actions are drawn from a mixture with substantially different coverage?

  3. What empirical mechanism connects one-step Action-NCE to the multi-step quantities used by CEM planning? Can the authors report horizon-dependent rollout error or calibration and analyze whether these quantities predict planning success, including under the official fixed-goal protocol where both methods obtain 0/250?

  4. In partially observed or stochastic transitions, the same latent transition may correspond to a multimodal posterior over actions rather than a single identifiable action. How does Action-NCE behave in such cases, and can the authors compare it with a probabilistic or multi-label inverse objective on an environment designed to contain genuinely ambiguous inverse dynamics?

Reviewer_LqdNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Action-Contrastive Masked Transition Modeling (AC-MTM), starting from LeWorldModel (LeWM). They remove the SIGReg regularizer and replace it with an Inverse Dynamics Modeling (IDM) loss. Their IDM head is trained with an InfoNCE objective (Action-NCE), contrary to previous very similar works like SMWM that use a regression IDM loss. AC-MTM is compared with a reproduction of LeWM and with inverse-dynamics regression models which they call MTM-MSE (from SMWM) on TwoRoom, Reacher, PushT and OGBench-Cube, plus a harder OGBench Visual Scene task, classical simulation tasks from LeWM used by many papers built on top of LeWM. AC-MTM roughly matches SIGReg on average over the four standard tasks and scores 80.0% vs. 58.0% on Scene, under a trajectory-goal protocol on which a random policy reaches 52%. It brings a gain on Reacher, OGB Scene and Cube but performs worse on Push-T.

Critical strengths
  1. Simple, cheap idea. Action-NCE is easy to implement, used only as an additional loss during training. It could be added on top of other end-to-end JEPA world models (LeWM, PLDM, H-JEPA) to measure when it helps.
  2. Clear core argument. The paper explains well why inverse regression can be a weak anti-collapse signal: under complete collapse, the best prediction is the mean action, so the loss is only bounded below by the action variance. Turning this into a chance-level classification floor is a reasonable fix.
  3. Some signal on Reacher. Inverse regression collapses on two of three seeds, while AC-MTM does not. This hints that the contrastive form can be more reliable, although the evidence is limited (see Weaknesses).
  4. Honest scope. The section "What Action-NCE does not solve", the PushT probes and the single-seed results in Appendix E correctly say that inverse-dynamics objectives favor agent-centric, controllable variables over weakly controlled object state. This is expected, and stating it openly is useful.
  5. Thorough protocol documentation. Appendices B and C record the evaluation choices, checkpoint selection and differences from the released LeWM code in detail, which helps reproducibility.
Critical weaknesses
  1. Limited novelty. Using inverse dynamics against collapse is not new here. The paper's inverse-regression baseline, MTM-MSE, is the inverse-dynamics MSE term already used in PLDM (Sobal et al., 2025) and hierarchical JEPA world models (H-JEPA, [citation]). Yet Table 1 describes PLDM only as VICReg variance/covariance, and the paper credits SMWM for this mechanism. SMWM already replaces SIGReg with inverse-action regression in the LeWM setting, and contrastive action or transition objectives exist in CLOUD, TACO and ACRO. The only change is MSE replaced by in-batch InfoNCE. The name "Masked Transition Modeling" overstates this: nothing is masked beyond ordinary joint forward and inverse dynamics training (as in Unified World Models), and the acronym MTM is already used by Masked Trajectory Models (Wu et al., ICML 2023). Eq. 5 is the standard log N floor of InfoNCE, not a new result.

  2. Slow-feature and partial collapse are ignored. A main reason to add an inverse-dynamics loss to JEPA world models is to counter slow-feature collapse (Sobal et al., 2022). PLDM and hierarchical JEPA world models (H-JEPA, [citation]) study this role; the paper cites neither line of work. Eq. 5 only rules out a constant encoder. The paper's own Appendix E shows the partial collapse that matters in practice: on Puzzle 4x4, AC-MTM decodes the button state at R² ≤ 0, versus R² ≈ 0.98 for SIGReg.

  3. The headline result sits close to chance. The 22-point Scene gain comes from a non-standard 25-step protocol where a random policy succeeds 52% of the time (one run, 95% CI 38.5 to 65.2%), so SIGReg's 58% is barely above random. Under the official OGBench protocol, both methods score 0/250. The proposed explanation, that the Gaussian prior becomes a bottleneck, is never tested: no per-factor probes are run, and the authors concede that lower forward error could explain the gap.

  4. Weak evidence for contrastive over regression. The case for Action-NCE over MTM-MSE rests on two collapsed Reacher seeds. Elsewhere MTM-MSE is equal or better (+3.8 on TwoRoom-long; Scene difference p ≈ 0.23). MTM-MSE uses a single fixed weight, whereas SMWM tunes it per task. There is no sweep over that weight, nor over λ, τ or the number of negatives, and no Action-NCE + SIGReg combination.

  5. Unconvincing gains and an unexplained baseline gap. On the standard suite, AC-MTM wins two tasks, ties one and loses PushT by 6.5 points. In Appendix E it loses Puzzle 4x4 and 4x5 by 16 and 26 points. This contradicts the claim that its advantage grows with the number of coupled controllable factors, since buttons are present in both Puzzle and Scene. Figure 3 also shows substantial differences between the reproduced SIGReg numbers and those reported in the LeWM paper; these are not explained. All results use three training seeds and a single evaluation seed.

Presentation:

  • Many formulations throughout the paper vague, imprecise and feel ai-generated ("offer an appealing route", "distills this line to its simplest form", "a distribution the dynamics never asked for", "localizes the credit")
  • The paper is inconsistent about MTM-MSE: Section 3 lists PushT among its strong tasks, while the introduction states that it trails SIGReg there (85.5% vs. 93.2%, Table 11).
  • The conclusion's claim of "preserving planning quality" is contradicted by the PushT and Puzzle results.
  • Notation is loose: "fwd_ϕ" and "inv_ψ" are used instead of single-letter maps, and p_i in Eq. 5 is not defined. -Figure 1 and 4 are raster images. Its CEM inference panel shows prior work, not a contribution (it could go to the appendix), and the orange arrow is unexplained.
  • The Planner column of Table 7 is constant. What is "AR" there ?
  • The bibliography (16 entries) is unusually short, which is not a good sign.
Questions for authors
  1. Can you explain the differences between your SIGReg results and those reported by LeWM? Was the SIGReg weight tuned per task?
  2. Does MTM-MSE still collapse on Reacher across a sweep of its loss weight?
  3. Can you report partial-collapse diagnostics (effective rank, per-factor probes on Scene) for all three objectives?
  4. PLDM and H-JEPA pair an MSE inverse-dynamics term with a variance/covariance regularizer and train stably. Does MTM-MSE still collapse on Reacher once a variance term is added? If not, the case for Action-NCE amounts to dropping that variance term rather than to the contrastive form being needed.
  5. What happens when Action-NCE is combined with SIGReg instead of replacing it?
  6. Can you report Scene results under a protocol where the random policy is well below 50%?

Comments on the paper

1 comment

We'd like to thank all reviewers for their careful feedback. They led us to strengthen the comparisons and correct the paper’s positioning.

  • LqdN — novelty and fair comparisons. We corrected attribution to existing inverse-dynamics and contrastive work, and added evaluation repeats, equal-budget tuning and stronger baselines. Inverse regression also works with tuning or a variance floor, Scene rankings change across evaluations, and the completed Reacher schedule comparison favors SIGReg. We no longer claim that the contrastive loss is necessary or consistently better (Appendix C.3, C.5, D.3–D.4).
  • gjWt — equivalent actions and long-term planning. We added measurements of action similarity, multi-step prediction errors and harder-goal evaluations (Appendix C.1–C.2, D.2). These measurements do not tell us how often different actions have the same effect or explain planning gains. Training that accepts equivalent actions, changes in data-collection policy and ambiguous action effects remain untested.
  • zE6c — sensitivity and missing state information. We added full-training batch/temperature tests and a combined Action-NCE + SIGReg study (Appendix D.1, C.4). The combination improved observed Scene planning and made some object states easier to read from the model than Action-NCE alone; rotation remained hard to read. Adding SIGReg does not answer how to retain all state without it.

What changed in the paper: We revised the abstract, introduction and conclusion, removed the unsupported explanation that Gaussian regularization caused the Scene deficit, and added full results and protocols. We distinguish our trajectory-goal tests from the official Scene task, which both methods fail.

Why we believe the paper merits acceptance: Action-NCE provides a simple training-only alternative to SIGReg with the same planner and test-time cost. Under the original ten-epoch recipe it improves TwoRoom and Cube, roughly matches Reacher and trails PushT. Its value is a usable method together with evidence about where it helps, where it fails and how it can complement regularization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.