- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose MotionJEPA: JEPA with a new regularizer DISReg. The motivation is that JEPA can encode backgrounds and other slowly changing features while losing information about the motion of important objects. With DISReg, the model learns to predict an embedding of the pixel difference between consecutive frames from their two representations. This encourages it to preserve information about changes between frames.
The main experiments use toy environments. Using high-capacity MLP probes, the authors show which features other methods lose and how MotionJEPA preserves them. Separate experiments show improved planning performance with static background distractors. The authors also probe the representations on real robot data from BridgeData V2.
Critical strengths
-
An interesting and clear idea: using frame differences to encourage the model to preserve information about dynamics.
-
A convincing evaluation of the idea. The controlled environments make it possible to examine which features are preserved and which are lost. The experiments include a Random baseline, different probes, ablations, and planning evaluations.
-
The paper is clearly written: the problem, the method, and the experimental findings are easy to follow.
Critical weaknesses
-
It is unclear how DISReg behaves with a moving camera and how robust it is to dynamic distractors. Dino includes moving clouds, but the experiments do not specifically test whether such changes can distract the model from important objects. A more detailed discussion of these limitations would be helpful.
-
The interpretation of Figure 3 needs further verification. Offline probes confirm poor final feature decodability, but it is unclear whether the information first becomes available and is then lost. The online probe may struggle to keep up with changes in the representation. It would be useful to train fresh probes on frozen intermediate checkpoints, at least for LeWM*.
Questions for authors
-
Does the pattern in Figure 3 for LeWM* persist when intermediate checkpoints are frozen and fresh probes are trained to convergence on each?
-
How does DISReg behave with a moving camera or dynamic distractors? Could the model primarily encode these changes while still ignoring important objects?
-
In Table 1b, Random outperforms several trained models on some features. How can the contribution of the dynamics predictor be separated from that of the high-capacity probe, which may itself learn to predict the state from the information retained in its input?
4 comments
W1 & Q2: Do gains persist with irrelevant visual changes?
MotionJEPA degrades under persistent or strong global changes, but tolerates them when they are mild or only partly present. We ran additional experiments with moving camera and illumination changes on Golf, probing the Encoder embedding (all runs are single-seed runs).
Moving Camera: We simulated a 2D moving camera by adding 10-pixel padding (from to image size) and then cropping a frame based on the camera's position (offset in pixels). Per episode, we sample a random initial camera position and movement direction with a fixed velocity (vel.). For reference, the average ball speed is 2.5 and bars 1.5 pixels per frame. The following table shows per-category probing losses for MotionJEPA + additional averages for LeWM* and LeWM-Flat*:
| vel. | avg. | ball | cross | bars | wood | cam | LeWM* (avg.) | LeWM-Flat* (avg.) |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.122 | 0.078 | 0.158 | 0.128 | 0.141 | 0.010 | 0.823 | 0.749 |
| 1 | 0.204 | 0.272 | 0.450 | 0.195 | 0.019 | 0.013 | 0.760 | 0.739 |
| 2 | 0.724 | 0.895 | 1.001 | 0.977 | 0.011 | 0.013 | 0.760 | 0.726 |
| 3 | 0.668 | 0.703 | 1.002 | 0.963 | 0.014 | 0.019 | 0.760 | 0.660 |
| 4 | 0.641 | 0.617 | 1.001 | 0.954 | 0.014 | 0.014 | 0.759 | 0.692 |
Note: velocity 0 still includes a random initial camera position, which adds variance to the scene, so losses are higher than in the standard Golf setup of Table 1. "avg." are weighted averages as in Table 1, not including the camera position.
We notice that MotionJEPA is sensitive to consistent camera movements, as the wood texture causes a lot of signal in every difference image. This results in MotionJEPA focusing on the wood texture (loss shrinks from 0.141 to below 0.02 with movement present), and the advantage disappears at 2 - 4 pixels per frame velocity due to suppressing the smaller dynamic features (ball, cross, bars). We also analyzed the impact of having a mix of episodes with moving (4 pixels per frame) and static camera for MotionJEPA:
| mov. | avg. | ball | cross | bars | wood | cam | LeWM* (avg.) | LeWM-Flat* (avg.) |
|---|---|---|---|---|---|---|---|---|
| 0% | 0.122 | 0.078 | 0.158 | 0.128 | 0.141 | 0.010 | 0.823 | 0.749 |
| 25% | 0.075 | 0.047 | 0.184 | 0.103 | 0.010 | 0.009 | 0.761 | 0.652 |
| 50% | 0.084 | 0.044 | 0.191 | 0.126 | 0.012 | 0.011 | 0.760 | 0.692 |
| 75% | 0.207 | 0.209 | 0.767 | 0.144 | 0.015 | 0.011 | 0.760 | 0.719 |
| 90% | 0.456 | 0.533 | 1.002 | 0.506 | 0.018 | 0.016 | 0.760 | 0.697 |
| 95% | 0.444 | 0.242 | 1.001 | 0.688 | 0.017 | 0.015 | 0.760 | 0.720 |
| 100% | 0.641 | 0.617 | 1.001 | 0.954 | 0.014 | 0.014 | 0.759 | 0.692 |
We observe that the performance of MotionJEPA stays similar up to 50% and then degrades gradually towards 100%, while still having better performance than LeWM-Flat* and LeWM* up to 95%, and similar performance at 100%. Notably, the wood probe already drops from 0.141 to 0.010 at 25% moving episodes, yet ball, cross, and bars remain well encoded. We therefore conclude that MotionJEPA is clearly better with mild or partial camera motion, and roughly on par with the baselines under persistent fast motion.
Illumination Changes: We animate gamma correction with a sine wave and report the impact of both frequency (cycles per frame) and amplitude (gamma correction range; 1.0 = no change):
| freq. | LeWM-Flat* (1.0) | LeWM-Flat* (0.9-1.1) | LeWM-Flat* (0.8-1.2) | LeWM-Flat* (0.7-1.3) | MotionJEPA (1.0) | MotionJEPA (0.9-1.1) | MotionJEPA (0.8-1.2) | MotionJEPA (0.7-1.3) |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.255 | 0.269 | 0.297 | 0.280 | 0.056 | 0.055 | 0.059 | 0.055 |
| 0.01 | - | 0.270 | 0.302 | 0.312 | - | 0.072 | 0.059 | 0.074 |
| 0.05 | - | 0.267 | 0.308 | 0.298 | - | 0.074 | 0.080 | 0.127 |
| 0.1 | - | 0.298 | 0.320 | 0.274 | - | 0.111 | 0.198 | 0.573 |
(LeWM* (not listed here) results are all in the range of 0.75 to 0.78)
A frequency of 0.1 cycles/frame means the scene goes from darkest to lightest in just 5 frames, which is a very fast change. We observe that MotionJEPA is stable for lower frequencies and amplitudes but highly sensitive to higher frequencies, with strong feature suppression at a frequency of 0.1 and gamma range 0.7 - 1.3, performing worse than LeWM-Flat*.
We add both the camera and illumination experiments to the appendix with more details in "Robustness to Large Visual Changes" and we updated the limitations section to reference and mention those as well. We are currently running additional planning experiments with object-type distractors and will add the results once available.
Dear Reviewer,
Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.
Thank you for your time and cooperation.
Q2 : Additional Moving-Distractor Results.
To further address your concern that moving distractors could cause MotionJEPA to ignore important scene elements, we have completed additional planning and frozen-state probing experiments across all four LeWM benchmarks. The results show improved planning performance to original LeWM and generally better preservation of positional information. The full results are provided in the new appendix paragraph “Moving Distractors.”
| Method | PushT (25 / 50) | TwoRoom (25 / 50) | Reacher (25 / 50) | Cube (25 / 50) | Mean (25 / 50) |
|---|---|---|---|---|---|
| LeWM | 2.4 / 0.4 | 33.6 / 16.0 | 8.4 / 6.8 | 43.2 / 27.6 | 21.9 / 12.7 |
| LeWM-Flat | 13.6 / 2.8 | 52.0 / 28.0 | 32.0 / 41.2 | 51.2 / 46.8 | 37.2 / 29.7 |
| MotionJEPA | 14.0 / 5.2 | 96.8 / 90.4 | 26.0 / 42.4 | 68.0 / 50.4 | 51.2 / 47.1 |
| MotionJEPA () | 67.6 / 30.8 | 99.6 / 80.4 | 29.6 / 49.6 | 62.8 / 47.2 | 64.9 / 52.0 |
Planning success rates (%), evaluated over 250 rollouts per setting. Mean is the arithmetic average across four environments. Each configuration uses one training run.
We thank the reviewer for the detailed review and address their concerns below
Offline probes on frozen checkpoints reproduce the dip in probing loss. We agree that training drift can cause online probes to underestimate performance. To reduce this risk, we use a significantly higher learning rate of 3e-3 (the same as used for offline probes) for the online probes vs. a learning rate of 5e-5 for training the JEPA models. The following table shows the Dinosaur's position probing for both online and offline probes, with snapshots taken around the local minimum (columns are training steps):
Both probe types reach their minimum at 5k and rise sharply afterwards, so the Figure 3 interpretation holds, as information is encoded early and then lost. Online probes report higher loss, as expected from fewer updates, but track the trajectory while giving finer temporal resolution at lower compute.
The high-capacity probes are identical for each method (same architecture, hyperparameters, and training budget). A probe only sees its input embedding and cannot add information that is not in it, so lower probing loss is a result of differences in the embeddings themselves. The main purpose of the high-capacity probe is to measure how much information is present in a given embedding. Random is not a competitor but a reference for the information available at initialization, as its weights are frozen. We therefore interpret a trained method that scores worse than Random as having lost information that was initially decodable.
To estimate the contribution of each component in preserving or increasing the encoded information, we compare trained baselines. For example, when probing the predicted rollout embedding of Dino, going from ForwardOnly (no regularization, 1.081) to LeWM* (SIGReg on z, 0.334) reduces the probing loss by about 0.747, which is the effect of SIGReg on the image embedding z. Going from LeWM* to MotionJEPA (DISReg, 0.055) reduces it by a further 0.279. DISReg keeps the same SIGReg term on z and adds the Diff Encoder and Diff Predictor with their additional losses, so this further reduction is the effect of the diff components.