- Rating
- 1
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper improves world action models for image goal-based planning and manipulation problem. Despite the strength of JEPAs, the authors raises three core challenges. To this end, the authors employ slot attention to JEPA world models. Experimental validation are performed on visual understanding tasks, planning tasks, and robot manipulation tasks.
Critical strengths
- Employing slot attention is reasonable
Critical weaknesses
-
Novelty clarification
- This paper misses several closely related works that shares similar ideas on the same problem. (e.g., Slot-MPC) [1]. The authors should clarify whats unique and superior than these previous works.
- Moreover, some relevant cited works are omit from quantitative comparison, such as Causal-JEPA [2] and Better Slots Better Worlds (SlotContrast-WM) [3]. These methods should be quantitatively compared in the experimental sections
-
Experimental validation is not sufficient. These should be at least one more popular benchmarks for each planning tasks and robot arm manipulation tasks.
-
The main results are limited to few planning and robot manipulation tasks, whose success rates are also high, mostly ranging beyond 80. These results cannot validate whether the proposed method still works well on challenging tasks that most models barely success.
-
In Table 3, quantitative comparison results on realm-jepa is done only with its base model. This is more like an ablation study rather than quantitative comparison with previous works.
[1] Slot-MPC, Spieler et al. [2] Causal-JEPA [3] Better
Questions for authors
Does the proposed method robust to the choice of K?
4 comments
Responses to W1 (part 2)
The following table directly compares Slot-JEPA with the closest object-centric dynamics methods. We group C-JEPA and SlotContrast-WM because SlotContrast-WM adopts the same OC-JEPA/C-JEPA predictor family.
| Aspect | C-JEPA / SlotContrast-WM | Slot-MPC | SlotFormer | Slot-JEPA |
|---|---|---|---|---|
| Slot decomposition | VideoSAUR/SlotContrast trained on the target environment | SAVi trained on the target environment | SAVi/STEVE trained on the target environment | Pretrained on generic videos and transferred without downstream retraining |
| Slot definition | K*T time-varying slot vectors encoding both semantics and dynamics | K*T time-varying slot vectors encoding both semantics and dynamics | K*T time-varying slot vectors encoding both semantics and dynamics | K semantic slot vector and K*T dynamic spatial masks |
| Prediction context | Contiguous observed slot history | Contiguous observed slot history | Contiguous burn-in slot history | Retrieved, non-contiguous transitions |
| Action-conditional prediction | Yes | Yes | No | Yes |
| Prediction loss | Masked l2 loss on slots | l2 loss on slots | l2 loss on slots | UOT and l1 losses over masks |
| Planning | Standard CEM with l2 loss | Gradient-based MPC or MPPI | Ranks predefined PHYRE actions | Standard CEM with UOT and l1 losses over masks |
Following your valuable suggestion, we also add quantitative comparisons on seven planar-planning tasks and the established OGBench robot-manipulation benchmark:
| Task Type | ID planar | ID planar | ID planar | OOD planar | OOD planar | OOD planar | ID planar | Robot manipulation |
|---|---|---|---|---|---|---|---|---|
| Method | TwoRoom (%) | Maze (%) | PushT (%) | TwoRoomR (%) | MazeR (%) | PushObj (%) | Reacher (%) | OGBench (%) |
| DINO-CLS | 6 | 88 | 20 | 34 | 70 | 2 | 84 | 68 |
| ResNet | 6 | 98 | 34 | 12 | 68 | 4 | 88 | 62 |
| R3M | 38 | 100 | 16 | 8 | 32 | 2 | 88 | 68 |
| DINO-WM | 50 | 100 | 74 | 82 | 86 | 28 | 82 | 84 |
| V-JEPA 2.1 | 50 | 94 | 58 | 50 | 86 | 46 | 88 | 80 |
| LeWM | 80 | 98 | 88 | 52 | 88 | 30 | 86 | 74 |
| C-JEPA | 92 | 98 | 80 | 40 | 52 | 28 | 82 | 72 |
| SlotContrast-WM | 78 | 84 | 62 | 40 | 58 | 18 | 70 | 70 |
| Slot-JEPA | 100 | 100 | 92 | 90 | 96 | 58 | 98 | 94 |
It can be seen that Slot-JEPA still outperforms these slot-based world models, especially on OOD tasks.
Responses to W2, W3, and W4
W2
One more popular benchmarks for planning and robot arm manipulation tasks.
Response to W2
We agree that evaluation on more benchmark is necessary, and update a standard planning benchmark Reacher and a robot manipulation benchmark OGBench, including the two closest object-centric world models (C-JEPA and SlotContrast-WM):
| Method | OGBench success (%) | Reacher success (%) |
|---|---|---|
| DINO-CLS | 68 | 84 |
| ResNet | 62 | 88 |
| R3M | 68 | 88 |
| DINO-WM | 84 | 82 |
| V-JEPA 2.1 | 80 | 88 |
| LeWM | 74 | 86 |
| C-JEPA | 72 | 82 |
| SlotContrast-WM | 70 | 70 |
| Slot-JEPA | 94 | 98 |
On these benchmarks, Slot-JEPA still outperforms previous JEPA world models, including slot-based methods.
W3
Success rates on planning and robot manipulation tasks are high, mostly ranging beyond 80.
Response to W3
Thank you for the suggestion. However, we would like to clarify that most JEPA-based methods do not achieve strong performance on the benchmarks used in our paper, and most success rates are below 80%. For example, on TwoRoom and PushT, some methods (such as DINO-WM) additionally receive proprioceptive goal-state information, including the target position and velocity of the moving object, rather than relying only on visual observations. After removing this additional information, the results are as follows:
| Method | TwoRoom (%) | PushT (%) | TwoRoomR (%) | PushObj (%) |
|---|---|---|---|---|
| DINO-CLS | 6 | 20 | 34 | 2 |
| ResNet | 6 | 34 | 12 | 4 |
| R3M | 38 | 16 | 8 | 2 |
| DINO-WM | 50 | 74 | 82 | 28 |
| V-JEPA 2.1 | 50 | 58 | 50 | 46 |
| LeWM | 80 | 88 | 52 | 30 |
| Slot-JEPA | 100 | 92 | 90 | 58 |
Without the additional low-dimensional goal state, most methods perform poorly, especially in the OOD settings TwoRoomR and PushObj. Slot-JEPA retains a clear advantage under this more challenging vision-only setting.
Additionally, REALM-JEPA is introduced precisely to provide a more challenging test platform for robot manipulation. Even V-JEPA 2.1, one of the strongest JEPA-based methods for robot manipulation, exhibits low success rates on this benchmark.
W4
In Table 3, quantitative comparison results on realm-jepa is done only with its base model.
Response to W4
Thank you for the suggestion. To the best of our knowledge, by the ICLR submission deadline, only two JEPA-based methods had been extended to this type of robot manipulation: V-JEPA 2.1 and Slot-MPC. However, Slot-MPC had not released the corresponding code or data. We therefore followed the evaluation protocol used by V-JEPA 2.1 for these tasks. We did not redesign predictors for other JEPA-based methods and transfer them to robot manipulation, because doing so would constitute an extension of those methods rather than a direct adaptation.
After the ICLR deadline, RoboJEPA was released in October. We therefore add RoboJEPA-8B as a competitor:
| Method | Reach success (%) | Pick success (%) | Rotate success (%) | Place success (%) |
|---|---|---|---|---|
| V-JEPA 2.1 | 72.2 | 63.3 | 20.0 | 0.0 |
| RoboJEPA-8B | 100.0 | 93.3 | 66.7 | 86.7 |
| Slot-JEPA | 100.0 | 90.0 | 83.3 | 76.7 |
RoboJEPA achieves higher success rates on Pick and Place but performs slightly worse on Rotate, which may result from the task-distribution bias in its training data. It is worth noting that RoboJEPA uses a ViT-G encoder and is trained on more than 15k video hours. In contrast, Slot-JEPA and V-JEPA 2.1 use a ViT-B encoder and train their robot predictor on DROID (414 video hours).
We have also added OGBench, a robot manipulation benchmark commonly used in prior work that features relatively simple scenes. Please see our response to W2.
Responses to Q1
Q1
Does the proposed method robust to the choice of K?
Response to Q1
By default, we set 8 slots for planar planning and 16 slots for REALM-JEPA. We conducte a sensitivity study with while keeping the encoder, predictor, training protocol, and planning budget fixed. We use the most challenging task from each benchmark: PushObj and Rotate.
| Task | (planar default) | (REALM-JEPA default) | |||
|---|---|---|---|---|---|
| PushObj (%) | 2 | 34 | 58 | 66 | 64 |
| Rotate (%) | 0 | 0 | 73.3 | 83.3 | 86.7 |
For the relatively simple PushObj scenes, is already sufficient to achieve strong performance. Increasing allows the model to decompose objects into finer-grained slots and can provide further gains, but this comes with increased computation time. On Rotate, performance largely saturates around , with only a marginal improvement from further increasing . In contrast, when is too small (e.g., ), the model can barely distinguish different objects and planning essentially fails. We therefore view the choice of K as a compute--performance trade-off: it should be selected according to the available computational budget, with a larger K preferred when resources permit.
Responses to W1 (part 1)
We sincerely thank the reviewer for the detailed and constructive feedback! We have carefully revised the manuscript in response to these comments and provide our point-by-point responses below.
W1
Misses several closely related works that shares similar ideas on the same problem; quantitative comparison with C-JEPA and SlotContrast-WM.
Response to W1
Thank you for bringing these relevant works to our attention! Because both “object-centric” and “world model” are broad concepts, these methods appear to address a similar topic but in fact focus on distinct questions and stages of the pipeline. They broadly follow the same pipeline: visual observations are converted into slots, a dynamics predictor models how the slots evolve, and the learned world model is used either for online planning or for policy learning. This gives four directions:
Compared with other object-centric dynamics prediction methods, Slot-JEPA has two main differences:
1. Task-agnostic decomposition. We argue that organizing patch-level representations into objects is independent of the actions and objectives of a downstream task. Therefore, the slot module should precede the action-conditioned predictor as a transferable component. We pretrain Slot-JEPA on Kinetics-400, freeze it across all downstream tasks, and train only the task-specific dynamics predictor. In contrast, previous methods still treat slot extraction as part of downstream-task training, and finetune it on the target environment.
2. Decoupling object semantics from spatial dynamics. Given T frames, previous methods reorganize patch-feature into K×T slot vectors, while ours use K shared vectros and K×T masks.
Sepcifically, Let N, K, and D denote the numbers of patches, slots, and feature dimensions, respectively. C-JEPA, SlotContrast-WM, Slot-MPC, and SlotFormer all follow the SAVi-style formulation: they map Zt∈RN×D to a complete slot state St∈RK×D and predict the evolution of the slot vectors:
\widehat S_{t+1}=F_\theta(S_{\le t},u_{\le t}),$$ Thus, appearance, position, and dynamics are entangled in each slot vector. Slot-JEPA instead extracts a shared semantic vectors $S\in\mathbb{R}^{K\times D}$ from reference features $Z_{\mathcal R}\in\mathbb{R}^{T_{\mathcal R}\times N\times D}$ and represents each frame by a spatial assignment $M_t\in[0,1]^{K\times N}$: $$S=G_\phi(Z_{\mathcal R}), \qquad M_t=D_\phi(Z_t,S), \qquad \widetilde z_{t,n}=\sum_k M_{t,k,n}s_k.$$ The predictor updates only $M_t$ while keeping $S$ fixed. Its main advantage is **temporal identity consistency**: the same object remains tied to the same slot rather than being represented by different slots over time, avoiding re-localization after it leaves the view or becomes occluded and reducing long-horizon matching difficulty. **This also enables slot-mask transition retrieval and displacement-aware mask-space planning.**