acceptodds
Under review as a conference paper at ICLR 2027

MotionJEPA: Preventing Temporal Feature Collapse by Capturing Visual Changes in Latent Space

22 views

Abstract

Joint Embedding Predictive Architectures (JEPAs) are a promising paradigm for learning task-agnostic latent world models without visual reconstruction. However, standard JEPA training exhibits a strong inductive bias towards slow features, causing feature suppression and the collapse of latent representation. While inverse dynamics provides temporal anti-collapse, it relies on action labels and offers little incentive to embed general, unlabeled dynamics. We introduce **D**ifference **I**mage and **S**ingle image embedding **Reg**ularization (DISReg), a novel regularizer that builds on an inverse-dynamics-style module that predicts temporal difference image embeddings without any pixel reconstruction loss, encouraging balanced static and dynamic feature learning. DISReg consists of a static term that shapes the distribution of the image embedding and encourages slow features, and a dynamic term, which, unlike direct regularization on the embedding, imposes no constraint on the image embedding's shape or distribution and instead only incentivizes that dynamic features be present. By integrating this regularizer into a standard JEPA, we establish our new architecture, MotionJEPA. Latent probing demonstrates that MotionJEPA produces more complete representations than other methods, and our trajectory analysis shows it maintains geometrically simple latent embeddings with low curvature. We further show that MotionJEPA improves downstream planning success under static-background distractors across four environments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Latent World Model, Feature Collapse, Computer Vision, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dksCOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose MotionJEPA: JEPA with a new regularizer DISReg. The motivation is that JEPA can encode backgrounds and other slowly changing features while losing information about the motion of important objects. With DISReg, the model learns to predict an embedding of the pixel difference between consecutive frames from their two representations. This encourages it to preserve information about changes between frames.

The main experiments use toy environments. Using high-capacity MLP probes, the authors show which features other methods lose and how MotionJEPA preserves them. Separate experiments show improved planning performance with static background distractors. The authors also probe the representations on real robot data from BridgeData V2.

Critical strengths
  1. An interesting and clear idea: using frame differences to encourage the model to preserve information about dynamics.

  2. A convincing evaluation of the idea. The controlled environments make it possible to examine which features are preserved and which are lost. The experiments include a Random baseline, different probes, ablations, and planning evaluations.

  3. The paper is clearly written: the problem, the method, and the experimental findings are easy to follow.

Critical weaknesses
  1. It is unclear how DISReg behaves with a moving camera and how robust it is to dynamic distractors. Dino includes moving clouds, but the experiments do not specifically test whether such changes can distract the model from important objects. A more detailed discussion of these limitations would be helpful.

  2. The interpretation of Figure 3 needs further verification. Offline probes confirm poor final feature decodability, but it is unclear whether the information first becomes available and is then lost. The online probe may struggle to keep up with changes in the representation. It would be useful to train fresh probes on frozen intermediate checkpoints, at least for LeWM*.

Questions for authors
  1. Does the pattern in Figure 3 for LeWM* persist when intermediate checkpoints are frozen and fresh probes are trained to convergence on each?

  2. How does DISReg behave with a moving camera or dynamic distractors? Could the model primarily encode these changes while still ignoring important objects?

  3. In Table 1b, Random outperforms several trained models on some features. How can the contribution of the dynamics predictor be separated from that of the high-capacity probe, which may itself learn to predict the state from the information retained in its input?

4 comments
AuthorsOpenReview ↗

We thank the reviewer for the detailed review and address their concerns below

Q1 Confirming LeWM* Online Probing Results with Offline Probes

Offline probes on frozen checkpoints reproduce the dip in probing loss. We agree that training drift can cause online probes to underestimate performance. To reduce this risk, we use a significantly higher learning rate of 3e-3 (the same as used for offline probes) for the online probes vs. a learning rate of 5e-5 for training the JEPA models. The following table shows the Dinosaur's position probing for both online and offline probes, with snapshots taken around the local minimum (columns are training steps):

Probe Type0.5k1k2k5k10k20k40k
Online0.4800.4210.2470.1180.3100.8830.704
Offline0.1380.1090.0710.0430.1530.7530.624

Both probe types reach their minimum at 5k and rise sharply afterwards, so the Figure 3 interpretation holds, as information is encoded early and then lost. Online probes report higher loss, as expected from fewer updates, but track the trajectory while giving finer temporal resolution at lower compute.

Q3 Separating Contribution of Dynamics Predictor vs. Offline Probe in Table 1b

The high-capacity probes are identical for each method (same architecture, hyperparameters, and training budget). A probe only sees its input embedding and cannot add information that is not in it, so lower probing loss is a result of differences in the embeddings themselves. The main purpose of the high-capacity probe is to measure how much information is present in a given embedding. Random is not a competitor but a reference for the information available at initialization, as its weights are frozen. We therefore interpret a trained method that scores worse than Random as having lost information that was initially decodable.

To estimate the contribution of each component in preserving or increasing the encoded information, we compare trained baselines. For example, when probing the predicted rollout embedding of Dino, going from ForwardOnly (no regularization, 1.081) to LeWM* (SIGReg on z, 0.334) reduces the probing loss by about 0.747, which is the effect of SIGReg on the image embedding z. Going from LeWM* to MotionJEPA (DISReg, 0.055) reduces it by a further 0.279. DISReg keeps the same SIGReg term on z and adds the Diff Encoder and Diff Predictor with their additional losses, so this further reduction is the effect of the diff components.

W1 & Q2: Do gains persist with irrelevant visual changes?

MotionJEPA degrades under persistent or strong global changes, but tolerates them when they are mild or only partly present. We ran additional experiments with moving camera and illumination changes on Golf, probing the Encoder embedding zz (all runs are single-seed runs).

Moving Camera: We simulated a 2D moving camera by adding 10-pixel padding (from 112×112112 \times 112 to 132×132132 \times 132 image size) and then cropping a 112×112112 \times 112 frame based on the camera's position (offset in pixels). Per episode, we sample a random initial camera position and movement direction with a fixed velocity (vel.). For reference, the average ball speed is 2.5 and bars 1.5 pixels per frame. The following table shows per-category probing losses for MotionJEPA + additional averages for LeWM* and LeWM-Flat*:

vel.avg.ballcrossbarswoodcamLeWM* (avg.)LeWM-Flat* (avg.)
00.1220.0780.1580.1280.1410.0100.8230.749
10.2040.2720.4500.1950.0190.0130.7600.739
20.7240.8951.0010.9770.0110.0130.7600.726
30.6680.7031.0020.9630.0140.0190.7600.660
40.6410.6171.0010.9540.0140.0140.7590.692

Note: velocity 0 still includes a random initial camera position, which adds variance to the scene, so losses are higher than in the standard Golf setup of Table 1. "avg." are weighted averages as in Table 1, not including the camera position.

We notice that MotionJEPA is sensitive to consistent camera movements, as the wood texture causes a lot of signal in every difference image. This results in MotionJEPA focusing on the wood texture (loss shrinks from 0.141 to below 0.02 with movement present), and the advantage disappears at 2 - 4 pixels per frame velocity due to suppressing the smaller dynamic features (ball, cross, bars). We also analyzed the impact of having a mix of episodes with moving (4 pixels per frame) and static camera for MotionJEPA:

mov.avg.ballcrossbarswoodcamLeWM* (avg.)LeWM-Flat* (avg.)
0%0.1220.0780.1580.1280.1410.0100.8230.749
25%0.0750.0470.1840.1030.0100.0090.7610.652
50%0.0840.0440.1910.1260.0120.0110.7600.692
75%0.2070.2090.7670.1440.0150.0110.7600.719
90%0.4560.5331.0020.5060.0180.0160.7600.697
95%0.4440.2421.0010.6880.0170.0150.7600.720
100%0.6410.6171.0010.9540.0140.0140.7590.692

We observe that the performance of MotionJEPA stays similar up to 50% and then degrades gradually towards 100%, while still having better performance than LeWM-Flat* and LeWM* up to 95%, and similar performance at 100%. Notably, the wood probe already drops from 0.141 to 0.010 at 25% moving episodes, yet ball, cross, and bars remain well encoded. We therefore conclude that MotionJEPA is clearly better with mild or partial camera motion, and roughly on par with the baselines under persistent fast motion.

Illumination Changes: We animate gamma correction with a sine wave and report the impact of both frequency (cycles per frame) and amplitude (gamma correction range; 1.0 = no change):

freq.LeWM-Flat* (1.0)LeWM-Flat* (0.9-1.1)LeWM-Flat* (0.8-1.2)LeWM-Flat* (0.7-1.3)MotionJEPA (1.0)MotionJEPA (0.9-1.1)MotionJEPA (0.8-1.2)MotionJEPA (0.7-1.3)
00.2550.2690.2970.2800.0560.0550.0590.055
0.01-0.2700.3020.312-0.0720.0590.074
0.05-0.2670.3080.298-0.0740.0800.127
0.1-0.2980.3200.274-0.1110.1980.573

(LeWM* (not listed here) results are all in the range of 0.75 to 0.78)

A frequency of 0.1 cycles/frame means the scene goes from darkest to lightest in just 5 frames, which is a very fast change. We observe that MotionJEPA is stable for lower frequencies and amplitudes but highly sensitive to higher frequencies, with strong feature suppression at a frequency of 0.1 and gamma range 0.7 - 1.3, performing worse than LeWM-Flat*.

We add both the camera and illumination experiments to the appendix with more details in "Robustness to Large Visual Changes" and we updated the limitations section to reference and mention those as well. We are currently running additional planning experiments with object-type distractors and will add the results once available.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Q2 : Additional Moving-Distractor Results.

To further address your concern that moving distractors could cause MotionJEPA to ignore important scene elements, we have completed additional planning and frozen-state probing experiments across all four LeWM benchmarks. The results show improved planning performance to original LeWM and generally better preservation of positional information. The full results are provided in the new appendix paragraph “Moving Distractors.”

MethodPushT (25 / 50)TwoRoom (25 / 50)Reacher (25 / 50)Cube (25 / 50)Mean (25 / 50)
LeWM2.4 / 0.433.6 / 16.08.4 / 6.843.2 / 27.621.9 / 12.7
LeWM-Flat13.6 / 2.852.0 / 28.032.0 / 41.251.2 / 46.837.2 / 29.7
MotionJEPA14.0 / 5.296.8 / 90.426.0 / 42.468.0 / 50.451.2 / 47.1
MotionJEPA (λpred=5\lambda_{pred}=5)67.6 / 30.899.6 / 80.429.6 / 49.662.8 / 47.264.9 / 52.0

Planning success rates (%), evaluated over 250 rollouts per setting. Mean is the arithmetic average across four environments. Each configuration uses one training run.

Reviewer_i82HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MotionJEPA, which addresses a failure mode in latent world models: preventing a representation from collapsing globally does not ensure that it retains the state variables needed for prediction and planning. The authors propose DISReg objective, which adds a learned embedding of the difference between consecutive images. As a result, a predictor must recover this difference embedding from the two corresponding image embeddings, while distribution regularization is applied to both image and difference embeddings. The paper tests this on synthetic games with known state variables, then evaluates planning under static distractors and adds a BridgeData analysis.

Critical strengths

The clearest contribution is the diagnosis of selective feature suppression. The synthetic environments expose individual state variables, so the evaluation can show what is lost rather than relying only on a global latent statistic or an attractive reconstruction. The comparison includes tuned baselines and modifications to regularization placement. This makes the result more informative than showing failure for a single default configuration.

The improvements in Table 1 are substantial. The reported three-run analysis strengthens this controlled evidence.

The paper connects the representation result to downstream behavior. In Table 3, standalone MotionJEPA reaches mean planning success of 81.8% and 70.3% in the two reported settings, compared with 77.2% and 59.3% for IDM. Combining the two objectives reaches 86.4% and 71.6%. The ablation is also informative which shows without the image-latent regularizer the system performs worse.

The BridgeData probes are also a useful attempt to move beyond the designed games. They provide some evidence that the effect is not confined to synthetic rendering, although the measured robot-state variables cover only part of what a real manipulation world model should represent.

Critical weaknesses

DISReg adds both a training target and additional trainable modules. The paper acknowledges the extra capacity and computation. However, the observation that baseline curves plateau with more training does not isolate these changes. More training steps are not equivalent to a different auxiliary network or a different source of supervision.

A control with comparable training capacity and an alternative auxiliary target would clarify how much of the gain comes specifically from difference-image prediction. It would also help to report parameter counts and wall-clock training costs.

The latent-path straightness analysis is interesting, but it is not a general measure of world-model quality. It uses interpolations of simulator state, and a representation can become geometrically simpler by removing information. MotionJEPA is also not uniformly the straightest model.

Likewise, a finite hyperparameter search cannot establish a fundamental inability of other anti-collapse objectives to retain the relevant variables. The search makes the empirical comparison stronger, but the conclusion should remain about the tested settings.

Questions for authors

What happens if the main controlled environments include camera motion or a moving distractor that is independent of the task state? I would like to see the same per-variable probes, together with planning performance, so that it is possible to distinguish retaining more visual change from retaining more useful information.

Can the authors compare against one auxiliary objective with a similar parameter and training-compute budget? You can also consider to report the added training cost. The fact that the auxiliary modules are removed at inference is valuable, but it answers a different question from the source of the training improvement.

4 comments
AuthorsOpenReview ↗

We thank the reviewer for the detailed feedback and address the concerns and questions below

Q1 Camera Motion or Moving Distractor

MotionJEPA degrades under persistent or strong global changes, but tolerates them when they are mild or only partly present. We ran the following additional experiments on Golf with probing of the Encoder embedding zz (all runs are single-seed runs).

We simulated a 2D moving camera by adding 10-pixel padding (from 112×112112 \times 112 to 132×132132 \times 132 image size) and then cropping a 112×112112 \times 112 frame based on the camera's position (offset in pixels). Per episode, we sample a random initial camera position and movement direction with a fixed velocity (vel.). (for reference, the average ball speed is 2.5 and bars 1.5 pixels per frame)

The following table shows per-category probing losses for MotionJEPA + additional averages for LeWM* and LeWM-Flat*:

vel.avg.ballcrossbarswoodcamLeWM* (avg.)LeWM-Flat* (avg.)
00.1220.0780.1580.1280.1410.0100.8230.749
10.2040.2720.4500.1950.0190.0130.7600.739
20.7240.8951.0010.9770.0110.0130.7600.726
30.6680.7031.0020.9630.0140.0190.7600.660
40.6410.6171.0010.9540.0140.0140.7590.692

Note: velocity 0 still includes a random initial camera position, which adds variance to the scene, so losses are higher than in the standard Golf setup of Table 1. Averages do not include the camera position.

We notice that MotionJEPA is sensitive to consistent camera movements, as the wood texture causes a lot of signal in every difference image. This results in MotionJEPA focusing on the wood texture (loss shrinks from 0.141 to below 0.02 with movement present), and the advantage disappears at 2 - 4 pixels per frame velocity due to suppressing the smaller dynamic features (ball, cross, bars). We then analyzed the impact of having a mix of episodes with moving (4 pixels per frame) and static camera for MotionJEPA:

mov.avg.ballcrossbarswoodcamLeWM* (avg.)LeWM-Flat* (avg.)
0%0.1220.0780.1580.1280.1410.0100.8230.749
25%0.0750.0470.1840.1030.0100.0090.7610.652
50%0.0840.0440.1910.1260.0120.0110.7600.692
75%0.2070.2090.7670.1440.0150.0110.7600.719
90%0.4560.5331.0020.5060.0180.0160.7600.697
95%0.4440.2421.0010.6880.0170.0150.7600.720
100%0.6410.6171.0010.9540.0140.0140.7590.692

We observe that the performance of MotionJEPA stays similar up to 50% and then degrades gradually towards 100%, while still having better performance than LeWM-Flat* and LeWM* up to 95%, and similar performance at 100%. Notably, the wood probe already drops from 0.141 to 0.010 at 25% moving episodes, yet ball, cross, and bars remain well encoded. We therefore conclude that MotionJEPA is clearly better with mild or partial camera motion, and roughly on par with the baselines under persistent fast motion.

We are currently running additional object-level distractor experiments for planning success rate on the LeWM Benchmarks and will also upload those as soon as possible. We added the "Robustness to large visual changes" section in the appendix with more details and also reference these limitations in our limitations section.

W1 & Q2 (1): Comparing against Other Auxiliary Objectives at same Compute Budget

With identical architecture, parameter count, and compute, replacing the difference image removes the improvement. We ran experiments on two auxiliary objectives for the Diff Encoder:

  • oto_t: Takes only the first observation.
  • Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1}): concatenates both observations along the channel dimension.

We perform a grid search on both λz∈{0.05,0.1,0.25,0.5,1}\lambda_z \in \{0.05, 0.1, 0.25, 0.5, 1\} and λd∈{0.5,1,2,5,10}\lambda_d \in \{0.5, 1, 2, 5, 10\} and then report the Encoder embedding zz probing loss for 3 seeded runs of the best hp set for the Golf environment (same protocol as Table 1). Regular MotionJEPA is the last row with the temporal difference image: ot+1−oto_{t+1} - o_t:

Aux. Targetλ_zλ_davg.ballcrossbarswood
oto_t0.2510.645 ± 0.0240.956 ± 0.0010.162 ± 0.0110.962 ± 0.0490.055 ± 0.026
Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1})0.520.606 ± 0.0200.929 ± 0.0310.192 ± 0.0100.870 ± 0.0450.049 ± 0.013
ot+1−oto_{t+1} - o_t0.2520.061 ± 0.0120.033 ± 0.0230.083 ± 0.0040.095 ± 0.0120.033 ± 0.008

Note: avg. is the weighted average over all probed variables, as in Table 1.

While having the same network structure, we can see that the oto_t and Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1}) perform significantly worse on dynamics such as the ball and bars, being close to a loss of 1.0. Since both targets are image-like, the regularization acts per frame and does not prevent collapse across time, and we hypothesize this leads to the same slow feature bias as in LeWM.

We also added a section in the appendix: "Contribution of the temporal difference image"

W1 & Q2 (2): Training Cost and Runtime vs. Training Improvement

For Table 1, we choose a high number of training samples to show that collapse is not an issue of low training budget. A full training run of 10M samples on game environments takes about 92 PFLOPs in 1.15 hours for LeWM (~15M parameters), and 160 PFLOPs in 1.6 hours for MotionJEPA (20.5M parameters). We trained MotionJEPA variants with varying compute budget on all three environments and report the average Encoder embedding zz probing loss of three seeds:

Method (train. set size)Pong (avg.)Dino (avg.)Golf (avg.)
LeWM-Flat* (10M)0.0080.3010.299
PLDM* (10M)0.1930.3530.373
MotionJEPA (10M)0.0040.0210.061
MotionJEPA (5.5M)0.0040.0220.088
MotionJEPA (2.5M)0.0050.0230.099
MotionJEPA (1M)0.0100.0300.172

MotionJEPA (5.5M) is slightly below LeWM's budget (~88 PFLOPs < 92 PFLOPs). We observe that MotionJEPA still outperforms both LeWM-Flat* and PLDM* on Dino and Golf, even with only 1M samples, which is about 17% of LeWM's compute budget.

More details are in the paper in the new section: "Model Size and Compute".

W2 Latent-path straightness is not a measure of world-model quality

Yes, we agree that straighter latent paths are not a general measure of world-model quality, which is why we prefer using non-linear MLP probes in Table 1 or downstream planning performance in Figure 5. We only present this as an additional analysis of MotionJEPA's representations, and as you mentioned, collapsing state variables can result in less curvature in the latent as well. We do not claim to have the straightest latent but want to show that introducing the DISReg tends to learn geometrically simpler representations, which can have practical advantages as listed in lines 381-385. Importantly, MotionJEPA’s straightness of latent paths is paired with low probing loss, meaning straightness is not obtained by discarding state variables.

W3 Hyperparameter search does not establish a fundamental limitation

We agree that a finite hyperparameter search cannot establish a theoretical or fundamental impossibility for other anti-collapse objectives to retain all relevant variables. We updated Section 4 (line 261) to remove claims of a 'fundamental limitation' and added more information on the scope of our conclusions to the evaluated architectures and searched hyperparameters in Appendix B.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Q1: Additional Experiments: Moving Distractors in LeWorldModel Benchmarks

To further address the concern about moving distractors, we conducted additional experiments across all four LeWorldModel benchmarks (PushT, TwoRoom, Reacher, and Cube) with independently moving visual distractors. We compare against both the original LeWM and LeWM-Flat, our modified baseline that applies SIGReg over the flattened batch and temporal dimensions instead of only the temporal dimension.

MethodPushT (25 / 50)TwoRoom (25 / 50)Reacher (25 / 50)Cube (25 / 50)Mean (25 / 50)
LeWM2.4 / 0.433.6 / 16.08.4 / 6.843.2 / 27.621.9 / 12.7
LeWM-Flat13.6 / 2.852.0 / 28.032.0 / 41.251.2 / 46.837.2 / 29.7
MotionJEPA14.0 / 5.296.8 / 90.426.0 / 42.468.0 / 50.451.2 / 47.1
MotionJEPA (λpred=5\lambda_{pred}=5)67.6 / 30.899.6 / 80.429.6 / 49.662.8 / 47.264.9 / 52.0

Planning success rates (%), evaluated over 250 rollouts per setting. Mean is the arithmetic average across four environments. Each configuration uses one training run.

MotionJEPA outperforms the original LeWM across all four environments. Compared with the stronger LeWM-Flat baseline, the default MotionJEPA achieves higher success in most settings, with particularly large gains on TwoRoom and Cube. Increasing the difference-prediction loss weight further improves PushT performance and 50-step success on Reacher.

The improved frozen-state probing results (see the new “Moving Distractors” paragraph in the appendix) further support better preservation of environment-state information, particularly positional information. Complete experimental details and probing results have been added to the appendix under the new paragraph “Moving Distractors.”

Reviewer_aPhwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MotionJEPA addresses feature suppression in JEPA world models, where predictable static content can dominate changing scene information. Its DISReg objective predicts a learned embedding of the difference image from consecutive state embeddings, alongside distribution regularization and forward prediction. The auxiliary objective requires no action labels. Synthetic probing, control experiments with static distractors, and BridgeData V2 probes show improved retention of dynamic information.

Critical strengths
  • The controlled analysis makes the feature-suppression problem clear. Independently sampled state factors, random-initialization baselines, nonlinear and linear probes, and probing of predicted rollouts provide complementary evidence. The baseline hyperparameter searches also strengthen the case that the observed failures are not simply poor default settings.
  • The method connects representation improvements to downstream control. It improves average planning success under static distractors and complements inverse-dynamics supervision. The auxiliary modules are discarded after training, so the improvement does not require additional inference modules.
Critical weaknesses
  • The experiments do not fully isolate the benefit of difference-image targets. DISReg adds both an auxiliary encoder and a prediction objective, while the main ablation removes the static regularization term. A matched auxiliary branch using individual-image targets would help establish whether explicitly encoding visual changes is essential to the gains.
  • The evidence for broader robustness remains limited. All control experiments use static wood-texture backgrounds, which largely cancel in frame differences. Performance with camera motion, illumination changes, or moving nuisance features remains unclear. BridgeData provides useful real-video evidence, but only through probes.
Questions for authors
  • How does DISReg compare with an auxiliary individual-image prediction objective using the same branch architecture and regularization?
  • Do the gains persist when irrelevant visual changes are introduced, such as moving backgrounds or illumination changes?
5 comments
AuthorsOpenReview ↗

We thank the reviewer for their detailed feedback and address the concerns below

W1 & Q1: Contribution of the Difference Image

The main improvement comes from the difference image, not solely the added modules & losses. We ran the suggested test and trained the same MotionJEPA architecture with two different auxiliary targets for the Diff Encoder:

  • oto_t: Takes only the first observation.
  • Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1}): concatenates both observations along the channel dimension.

We perform a grid search on both λz∈{0.05,0.1,0.25,0.5,1}\lambda_z \in \{0.05, 0.1, 0.25, 0.5, 1\} and λd∈{0.5,1,2,5,10}\lambda_d \in \{0.5, 1, 2, 5, 10\} and then report the Encoder embedding zz probing loss for 3 seeded runs of the best hp set for the Golf environment (same protocol as Table 1). Regular MotionJEPA is the last row with the temporal difference image: ot+1−oto_{t+1} - o_t:

Aux. Targetλ_zλ_davg.ballcrossbarswood
oto_t0.2510.645 ± 0.0240.956 ± 0.0010.162 ± 0.0110.962 ± 0.0490.055 ± 0.026
Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1})0.520.606 ± 0.0200.929 ± 0.0310.192 ± 0.0100.870 ± 0.0450.049 ± 0.013
ot+1−oto_{t+1} - o_t0.2520.061 ± 0.0120.033 ± 0.0230.083 ± 0.0040.095 ± 0.0120.033 ± 0.008

While having the same network structure, we can see that the oto_t and Concat(ot,ot+1)\text{Concat}(o_t, o_{t+1}) perform significantly worse on dynamics such as the ball and bars, being close to a loss of 1.0. By encoding regular images, the Diff Encoder is similar to the image Encoder, we hypothesize that this leads to the same slow feature bias as present in LeWM.

We also added these results in the appendix: "Contribution of the Temporal Difference Image"

W2 & Q2: Do gains persist with irrelevant visual changes?

MotionJEPA degrades under persistent or strong global changes, but tolerates them when they are mild or only partly present. We did additional experiments with a moving camera (which also results in background movement) and illumination changes. All experiments are single runs on Golf, and we report probing losses of the Encoder embeddings zz.

Moving Camera: We simulated a 2D moving camera by adding 10-pixel padding (from 112×112112 \times 112 to 132×132132 \times 132 image size) and then cropping a 112×112112 \times 112 frame based on the camera's position (offset in pixels). Per episode, we sample a random initial camera position and movement direction with a fixed velocity (vel.). For reference, the average ball speed is 2.5, and bars are 1.5 pixels per frame:

vel. (pixels / frame)LeWM* (avg.)LeWM-Flat* (avg.)MotionJEPA (avg.)
00.8230.7490.122
10.7600.7390.204
20.7600.7260.724
30.7600.6600.668
40.7590.6920.641

Note: velocity 0 still includes a random initial camera position, which adds variance to the scene, so losses are higher than in the standard Golf setup of Table 1.

We notice that MotionJEPA is sensitive to persistent camera movements, as the wood texture causes a lot of signal in every difference image, and the advantage disappears from 2 - 4 pixels per frame velocity. We further analyzed the impact of having a mix of episodes with moving (4 pixels per frame) and static camera:

movingLeWM* (avg.)LeWM-Flat* (avg.)MotionJEPA (avg.)
0%0.8230.7490.122
25%0.7610.6520.075
50%0.7600.6920.084
75%0.7600.7190.207
90%0.7600.6970.456
95%0.7600.7200.444
100%0.7590.6920.641

We observe that the performance of MotionJEPA stays similar up to 50% and then degrades gradually towards 100%, while still having better performance than LeWM-Flat* and LeWM* up to 95%.

Illumination Changes: For illumination changes, we animate gamma correction with a sine wave and report the impact of both frequency (cycles per frame) and amplitude (gamma correction range; 1.0 = no change):

freq.LeWM-Flat* (1.0)LeWM-Flat* (0.9-1.1)LeWM-Flat* (0.8-1.2)LeWM-Flat* (0.7-1.3)Ours (1.0)Ours (0.9-1.1)Ours (0.8-1.2)Ours (0.7-1.3)
00.2550.2690.2970.2800.0560.0550.0590.055
0.01-0.2700.3020.312-0.0720.0590.074
0.05-0.2670.3080.298-0.0740.0800.127
0.1-0.2980.3200.274-0.1110.1980.573

(LeWM* (not listed here) results are all in the range of 0.75 to 0.78)

A frequency of 0.1 cycles/frame means the scene goes from darkest to lightest in just 5 frames, which is a very fast change. We observe that MotionJEPA is stable for lower frequencies and amplitudes but highly sensitive to higher frequencies, with strong feature suppression at a frequency of 0.1 and gamma range 0.7 - 1.3, performing worse than LeWM-Flat*.

We add both the camera and illumination experiments to the appendix with more details in "Robustness to Large Visual Changes" and we updated the limitations section to reference and mention those as well.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Changes

We thank the reviewers for their valuable feedback, which has helped us improve the manuscript. We uploaded a revision with new results and clarifications (changes marked in red with strikethroughs showing previous text in the PDF):

  • (Pages 27, 28): New ablation, replacing temporal difference image with regular observations (Table 11), showing that the difference image is crucial for encoding dynamics (0.6450.645 and 0.606→0.0610.606 \rightarrow 0.061 on Golf environment).
  • (Pages 25, 26): New experiments on moving camera (Figure 14, Table 8), MotionJEPA outperforms LeWM and LeWM-Flat on mild or partly present camera movement, matching on strong global movements.
  • (Page 26): New experiments on illumination changes (Table 9), MotionJEPA remains stable and outperforms baselines under low to moderate frequencies & amplitudes.
  • (Pages 27, 28): New experiments add a dynamic distractor (Figure 15, Table 10) and shows MotionJEPA outperforming LeWM and LeWM-Flat baselines on average across the four benchmarks (LeWM: 21.9, LeWM-Flat: 37.2, MotionJEPA: 51.2)
  • (Pages 27 - 29): New experiments training MotionJEPA on matching and lower compute budget (Table 12), showing that strong results are not due to higher training compute. We also added analysis of model size and training runtime / compute budget.
  • (Pages 5, 14): Clarified scope of hyperparameter grid search regarding baseline limitations.
  • (Pages 9, 10): Updated limitations to refer to new experiments regarding sensitivity to large visual changes.
  • (Pages 34 - 36): Added per-category standard deviation of MotionJEPA lower compute variants in the per environment tables.

We remain available in case of any further requests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.