acceptodds
Withdrawn from ICLR 2027

Motion-Guided Contrastive Regularization for Latent World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

JEPA-style world models learn action-conditioned latent dynamics without reconstructing observations, but distributional regularization alone does not ensure that their representations preserve information needed for control. We identify static collapse, a failure mode in which the encoder preserves episode-specific static content while suppressing manipulation-relevant state changes. Because static content is temporally predictable yet diverse across episodes, this failure can persist despite regularizers such as SIGReg. A controlled study on OGBench-Cube shows that introducing randomized backgrounds substantially degrades planning performance while leaving the underlying manipulation task unchanged. To address this problem, we propose motion-guided contrastive regularization. Our approach compares predicted latent transitions with transitions to candidate observations from the same episode, reducing the usefulness of static background identity for distinguishing action outcomes. Optical-flow similarities provide soft contrastive targets, allowing candidates with similar motion to receive similar supervision. Combined with latent prediction and SIGReg, our objective encourages representations to preserve within-episode state distinctions and similarities between related transitions. Our method improves planning success over LeWorldModel (LeWM) from 52% to 78% on OGBench-Cube with randomized backgrounds and from 14% to 54% on LIBERO-Object, while maintaining comparable performance on four other visual-control benchmarks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent world models, joint-embedding predictive architectures (JEPA), representation collapse, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4m1EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a failure mode in which latent world models retain static background variation while underrepresenting actionable dynamics. It proposes motion-guided contrastive regularization: optical-flow distances define soft targets over within-episode candidate transitions, and predicted latent changes are trained to match them. Optical flow is computed offline and is unnecessary during planning. Experiments include a controlled background intervention, planning benchmarks, physical-state probes, and matched contrastive ablations.

Critical strengths
  1. The paired Cube/CubeRB intervention and state probes provide unusually direct evidence for the proposed diagnosis. Success changes from 52% to 78% on CubeRB, while robot-position probe error decreases from 13.01 to 3.56 cm. The clean Cube result changes only from 74% to 76%, supporting a targeted benefit under background distraction.

  2. Table 3 compares motion targets with InfoNCE using the same candidate construction and latent scores. The proposed targets improve CubeRB/LIBERO success to 78%/54%, versus 56%/34% for InfoNCE. The shared planner and absence of optical flow at inference make this a useful training-time intervention.

Critical weaknesses
  1. The statistical evidence is limited to 50 evaluation pairs per setting. LIBERO uses 50 pairs in total across ten tasks and a custom positional image-goal criterion; this is not equivalent to the original task predicates involving other state attributes. Per-task results, uncertainty estimates, and a clear account of initially satisfied Cube goals would better establish how broadly the reported gains apply.

  2. The training cost and dependence on pretrained optical flow are insufficiently quantified. Inference timing omits flow precomputation and additional candidate encodings during training. The InfoNCE control establishes the benefit of motion-shaped targets, but does not determine whether the pretrained flow model is necessary relative to a cheaper motion proxy with similarly soft targets. These issues affect the practicality and interpretation of the intervention.

Questions for authors
  1. Please report LIBERO results by task, clarifying the distinction between positional goal success and original task success. How do Cube/CubeRB results change when initially satisfied goals are excluded?

  2. What are flow precomputation time, training wall-clock time, and peak memory relative to LeWM? Can a cheaper motion proxy with matched soft-target construction recover the gains?

Reviewer_g8R1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies static collapse in end-to-end JEPA world models, representations retain episode-specific backgrounds while losing manipulation-relevant state information. It augments LeWM with a within-episode contrastive loss over latent transitions, using optical-flow similarities as soft targets alongside prediction loss and SIGReg. Reported image-goal planning success improves from 52% to 78% on Cube-RB and from 14% to 54% on LIBERO-Object. Reconstruction, attention, physical-state probes, and objective ablations provide supporting evidence. The evaluation concerns short-horizon goal-state matching; the LIBERO result is not the original benchmark’s task-completion rate.

Critical strengths
  1. The Cube/Cube-RB comparison is a well-motivated intervention, background appearance changes while manipulation dynamics remain fixed. The state probes provide quantitative evidence beyond visualization; for example, Cube-RB cube-position error decreases from 10.49 cm to 5.10 cm (Table 2).
  2. The proposed addition is simple and produces sizeable reported gains over the LeWM backbone. Table 3 includes a useful matched-candidate InfoNCE baseline, with gains of 22 and 20 percentage points from motion-guided targets on Cube-RB and LIBERO-Object. Offline flow computation leaves the deployed planner unchanged.
Critical weaknesses
  1. The evaluation reporting does not yet support a reliable comparison. Appendix A.2 states that each setting uses 50 binary evaluation instances, yet Table 1 reports PLDM results of 97% on 2-Rooms and 65% on Cube, and SMWM results of 83% on Push-T and 99% on 2-Rooms. Such values cannot be raw success fractions over 50 instances, whose resolution is two percentage points. Averaging or another denominator could explain this, but neither is documented. Training-run variability and uncertainty are also unreported. This matters particularly for the advantage over SMWM: the methods tie on Cube-RB, and 54% versus 40% on LIBERO-Object represents only seven additional successes under the stated denominator. The large gains over LeWM are promising, but the comparison needs an auditable aggregation protocol and repeated runs.
  2. The ablation supports the full objective but does not isolate the value of motion-derived target structure. Table 3 compares one-hot InfoNCE with flow-derived soft targets, changing both the information source and target softness. It does not exclude generic label smoothing or simpler similarity targets as explanations for the improvement. This distinction is central because within-trajectory negatives and flow-similarity supervision have prior precedents, which the paper appropriately cites (Gulati and Nemenman, 2026; Mahendran et al., 2018). A matched-softness control is needed to establish the contribution of the proposed motion-based weighting beyond these ingredients.
Questions for authors
  1. Clarification and verification: What are the exact denominators, training/evaluation seed counts, and aggregation rules for each Table 1 entry? Please reconcile the non-even percentages with Appendix A.2 and report repeated-run uncertainty for the principal comparisons. Were the hyperparameters in Appendix B selected on a validation set independent of the reported evaluation pairs?
  2. Targeted evidence: With candidates, loss weights, and tuning budgets matched, does the method outperform (i) entropy-matched label-smoothed InfoNCE and (ii) a control that retains the positive target mass but permutes the soft weights among alternative candidates? These tests would determine whether meaningful candidate-specific motion information, rather than softness alone, explains the gain.
Reviewer_uV7XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies static collapse in JEPA-style world models, in which representations retain episode-specific appearance but lose manipulation-relevant variation. It adds a contrastive objective that compares predicted latent changes with transitions to observations from the same episode.

A pretrained optical-flow model supplies soft similarity targets. The method is evaluated through image-goal planning in six settings, physical-state probes, reconstruction analyses, and objective ablations.

Critical strengths

The Cube/Cube-RB comparison is a useful diagnostic because it changes the background while keeping the physical trajectories and actions fixed. The representation analysis also goes beyond t-SNE: Table 2 reports a reduction in cube-position error from 10.49 cm to 5.10 cm on Cube-RB. This supports the claim that the learned representation makes manipulation state more accessible.

There is a meaningful improvement over the LeWM backbone under the paper’s planning protocol, particularly on LIBERO-Object. The appendix also makes clear that evaluation demonstrations are withheld before generating perturbed training trajectories, and that optical-flow computation is performed offline. These are sensible choices for evaluating a training-time regularizer without changing the deployed planner. 43596_Motion_Guided_Contrastiv

Critical weaknesses

I am not yet convinced that motion information explains the gain over ordinary contrastive training. Section 2 already identifies close precedents for within-trajectory negatives and flow-derived similarity targets, making the benefit of their combination important to establish. Table 3 compares hard InfoNCE targets with motion-derived soft targets, changing both their information content and their softness. An entropy-preserving shuffle of the negative target weights, with the positive weight fixed, would help distinguish meaningful motion supervision from a smoothing effect.
I also cannot reconcile Table 1 with the stated evaluation counts. Appendix A.2 specifies 50 binary success evaluations per setting, which would give percentages in increments of two. Yet the table includes baseline results of 83%, 97%, and 99%. These may be averages across runs, but no such averaging or training-seed variability is reported. This needs clarification before the table can substantiate the claimed common-protocol comparison

Questions for authors
  • Does the advantage over InfoNCE persist when the positive target probability and target entropy are preserved, but motion-derived weights are permuted among the negative candidates?
  • What are the trial counts, number of independently trained models, and aggregation rule for each row of Table 1? Please explain how the odd-numbered percentages arise under the stated 50-instance protocol and report the corresponding variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.