acceptodds
Under review as a conference paper at ICLR 2027

Slot-JEPA: Object-Structured Prediction Interfaces for Latent World Models

Abstract

As a prototypical latent world model, Joint-Embedding Predictive Architectures (JEPAs) provide strong semantic representations. However, dense patch representations expose a redundant interface to downstream predictors, which must implicitly infer spatial organization and temporal correspondence, presenting three challenges. First, object information is distributed across patches, forcing each downstream predictor to redundantly recover object structure that could otherwise be reused across tasks before modeling interactions. Second, the lack of explicit entity correspondence across time makes historical transitions expensive, restricting the predictor's flexibility in utilizing context. Third, limited by the lack of explicit transport structure, existing JEPA models typically use pointwise distances, which are insensitive to spatial displacement when predicted and goal supports are largely disjoint. We propose the Slot-based Joint-Embedding Predictive Architecture (Slot-JEPA), which decomposes dense representations into stable semantic slots and time-varying spatial masks. Pretrained once and frozen downstream, this decomposition exposes a reusable object-structured interface. Building on it, our Transition Memory Predictor (TMP) stores visual transitions as compact masks and retrieves useful context using statistics of corresponding slots. Its two branches separate slot-specific dynamics from shared cross-slot dynamics. We further use Unbalanced Optimal Transport (UOT) to supervise mask prediction and evaluate candidate plans by transporting mass between aligned predicted and goal masks, providing a signal better aligned with spatial progress. Across seven visual-understanding benchmarks, the slot-reconstructed features remain transferable. Slot-JEPA also improves goal-conditioned planning across six planar environments and four robotic arm planning tasks on the REALM-JEPA benchmark.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual World Models, Representation Learning, Joint-Embedding Predictive Architectures, Object-Centric Representation Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_9NK8OpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper improves world action models for image goal-based planning and manipulation problem. Despite the strength of JEPAs, the authors raises three core challenges. To this end, the authors employ slot attention to JEPA world models. Experimental validation are performed on visual understanding tasks, planning tasks, and robot manipulation tasks.

Critical strengths
  • Employing slot attention is reasonable
Critical weaknesses
  • Novelty clarification

    • This paper misses several closely related works that shares similar ideas on the same problem. (e.g., Slot-MPC) [1]. The authors should clarify whats unique and superior than these previous works.
    • Moreover, some relevant cited works are omit from quantitative comparison, such as Causal-JEPA [2] and Better Slots Better Worlds (SlotContrast-WM) [3]. These methods should be quantitatively compared in the experimental sections
  • Experimental validation is not sufficient. These should be at least one more popular benchmarks for each planning tasks and robot arm manipulation tasks.

  • The main results are limited to few planning and robot manipulation tasks, whose success rates are also high, mostly ranging beyond 80. These results cannot validate whether the proposed method still works well on challenging tasks that most models barely success.

  • In Table 3, quantitative comparison results on realm-jepa is done only with its base model. This is more like an ablation study rather than quantitative comparison with previous works.

[1] Slot-MPC, Spieler et al. [2] Causal-JEPA [3] Better

Questions for authors

Does the proposed method robust to the choice of K?

4 comments
AuthorsOpenReview ↗

Responses to W1 (part 1)

We sincerely thank the reviewer for the detailed and constructive feedback! We have carefully revised the manuscript in response to these comments and provide our point-by-point responses below.

W1

Misses several closely related works that shares similar ideas on the same problem; quantitative comparison with C-JEPA and SlotContrast-WM.

Response to W1

Thank you for bringing these relevant works to our attention! Because both “object-centric” and “world model” are broad concepts, these methods appear to address a similar topic but in fact focus on distinct questions and stages of the pipeline. They broadly follow the same pipeline: visual observations are converted into slots, a dynamics predictor models how the slots evolve, and the learned world model is used either for online planning or for policy learning. This gives four directions:

DirectionCore questionRepresentative methodsRelation to Slot-JEPA
Object-Centric Representation LearningHow are visual observations converted into slots?DINOSAUR, SIMONe, SAVi, SlotContrastRelated representation-learning precedents; Slot-JEPA trains its own decomposition module on V-JEPA 2.1 features
Object-Centric Dynamics PredictionHow does the object-centric state evolve over time and, when available, under actions?C-JEPA, SlotContrast-WM, SlotFormer, Slot-MPCSlot-JEPA belongs to this category
Object-Centric Model-Based PlanningHow is the learned model used to select actions online?OP3, STOVEUses predicted rollouts for direct planning
Object-Centric Model-Based RLHow is the learned model used to train behavior policies or exploration strategies?FOCUS, SOLDUses object-centric rollouts or rewards for policy learning

Compared with other object-centric dynamics prediction methods, Slot-JEPA has two main differences:

1. Task-agnostic decomposition. We argue that organizing patch-level representations into objects is independent of the actions and objectives of a downstream task. Therefore, the slot module should precede the action-conditioned predictor as a transferable component. We pretrain Slot-JEPA on Kinetics-400, freeze it across all downstream tasks, and train only the task-specific dynamics predictor. In contrast, previous methods still treat slot extraction as part of downstream-task training, and finetune it on the target environment.

2. Decoupling object semantics from spatial dynamics. Given T frames, previous methods reorganize patch-feature into K×TK\times T slot vectors, while ours use KK shared vectros and K×TK\times T masks.

Sepcifically, Let NN, KK, and DD denote the numbers of patches, slots, and feature dimensions, respectively. C-JEPA, SlotContrast-WM, Slot-MPC, and SlotFormer all follow the SAVi-style formulation: they map Zt∈RN×DZ_t\in\mathbb{R}^{N\times D} to a complete slot state St∈RK×DS_t\in\mathbb{R}^{K\times D} and predict the evolution of the slot vectors:

\widehat S_{t+1}=F_\theta(S_{\le t},u_{\le t}),$$ Thus, appearance, position, and dynamics are entangled in each slot vector. Slot-JEPA instead extracts a shared semantic vectors $S\in\mathbb{R}^{K\times D}$ from reference features $Z_{\mathcal R}\in\mathbb{R}^{T_{\mathcal R}\times N\times D}$ and represents each frame by a spatial assignment $M_t\in[0,1]^{K\times N}$: $$S=G_\phi(Z_{\mathcal R}), \qquad M_t=D_\phi(Z_t,S), \qquad \widetilde z_{t,n}=\sum_k M_{t,k,n}s_k.$$ The predictor updates only $M_t$ while keeping $S$ fixed. Its main advantage is **temporal identity consistency**: the same object remains tied to the same slot rather than being represented by different slots over time, avoiding re-localization after it leaves the view or becomes occluded and reducing long-horizon matching difficulty. **This also enables slot-mask transition retrieval and displacement-aware mask-space planning.**
AuthorsOpenReview ↗

Responses to W1 (part 2)

The following table directly compares Slot-JEPA with the closest object-centric dynamics methods. We group C-JEPA and SlotContrast-WM because SlotContrast-WM adopts the same OC-JEPA/C-JEPA predictor family.

AspectC-JEPA / SlotContrast-WMSlot-MPCSlotFormerSlot-JEPA
Slot decompositionVideoSAUR/SlotContrast trained on the target environmentSAVi trained on the target environmentSAVi/STEVE trained on the target environmentPretrained on generic videos and transferred without downstream retraining
Slot definitionK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK semantic slot vector and K*T dynamic spatial masks
Prediction contextContiguous observed slot historyContiguous observed slot historyContiguous burn-in slot historyRetrieved, non-contiguous transitions
Action-conditional predictionYesYesNoYes
Prediction lossMasked l2 loss on slotsl2 loss on slotsl2 loss on slotsUOT and l1 losses over masks
PlanningStandard CEM with l2 lossGradient-based MPC or MPPIRanks predefined PHYRE actionsStandard CEM with UOT and l1 losses over masks

Following your valuable suggestion, we also add quantitative comparisons on seven planar-planning tasks and the established OGBench robot-manipulation benchmark:

Task TypeID planarID planarID planarOOD planarOOD planarOOD planarID planarRobot manipulation
MethodTwoRoom (%)Maze (%)PushT (%)TwoRoomR (%)MazeR (%)PushObj (%)Reacher (%)OGBench (%)
DINO-CLS68820347028468
ResNet69834126848862
R3M381001683228868
DINO-WM50100748286288284
V-JEPA 2.15094585086468880
LeWM8098885288308674
C-JEPA9298804052288272
SlotContrast-WM7884624058187070
Slot-JEPA100100929096589894

It can be seen that Slot-JEPA still outperforms these slot-based world models, especially on OOD tasks.

AuthorsOpenReview ↗

Responses to W2, W3, and W4

W2

One more popular benchmarks for planning and robot arm manipulation tasks.

Response to W2

We agree that evaluation on more benchmark is necessary, and update a standard planning benchmark Reacher and a robot manipulation benchmark OGBench, including the two closest object-centric world models (C-JEPA and SlotContrast-WM):

MethodOGBench success (%)Reacher success (%)
DINO-CLS6884
ResNet6288
R3M6888
DINO-WM8482
V-JEPA 2.18088
LeWM7486
C-JEPA7282
SlotContrast-WM7070
Slot-JEPA9498

On these benchmarks, Slot-JEPA still outperforms previous JEPA world models, including slot-based methods.

W3

Success rates on planning and robot manipulation tasks are high, mostly ranging beyond 80.

Response to W3

Thank you for the suggestion. However, we would like to clarify that most JEPA-based methods do not achieve strong performance on the benchmarks used in our paper, and most success rates are below 80%. For example, on TwoRoom and PushT, some methods (such as DINO-WM) additionally receive proprioceptive goal-state information, including the target position and velocity of the moving object, rather than relying only on visual observations. After removing this additional information, the results are as follows:

MethodTwoRoom (%)PushT (%)TwoRoomR (%)PushObj (%)
DINO-CLS620342
ResNet634124
R3M381682
DINO-WM50748228
V-JEPA 2.150585046
LeWM80885230
Slot-JEPA100929058

Without the additional low-dimensional goal state, most methods perform poorly, especially in the OOD settings TwoRoomR and PushObj. Slot-JEPA retains a clear advantage under this more challenging vision-only setting.

Additionally, REALM-JEPA is introduced precisely to provide a more challenging test platform for robot manipulation. Even V-JEPA 2.1, one of the strongest JEPA-based methods for robot manipulation, exhibits low success rates on this benchmark.

W4

In Table 3, quantitative comparison results on realm-jepa is done only with its base model.

Response to W4

Thank you for the suggestion. To the best of our knowledge, by the ICLR submission deadline, only two JEPA-based methods had been extended to this type of robot manipulation: V-JEPA 2.1 and Slot-MPC. However, Slot-MPC had not released the corresponding code or data. We therefore followed the evaluation protocol used by V-JEPA 2.1 for these tasks. We did not redesign predictors for other JEPA-based methods and transfer them to robot manipulation, because doing so would constitute an extension of those methods rather than a direct adaptation.

After the ICLR deadline, RoboJEPA was released in October. We therefore add RoboJEPA-8B as a competitor:

MethodReach success (%)Pick success (%)Rotate success (%)Place success (%)
V-JEPA 2.172.263.320.00.0
RoboJEPA-8B100.093.366.786.7
Slot-JEPA100.090.083.376.7

RoboJEPA achieves higher success rates on Pick and Place but performs slightly worse on Rotate, which may result from the task-distribution bias in its training data. It is worth noting that RoboJEPA uses a ViT-G encoder and is trained on more than 15k video hours. In contrast, Slot-JEPA and V-JEPA 2.1 use a ViT-B encoder and train their robot predictor on DROID (414 video hours).

We have also added OGBench, a robot manipulation benchmark commonly used in prior work that features relatively simple scenes. Please see our response to W2.

AuthorsOpenReview ↗

Responses to Q1

Q1

Does the proposed method robust to the choice of K?

Response to Q1

By default, we set 8 slots for planar planning and 16 slots for REALM-JEPA. We conducte a sensitivity study with K∈{2,4,8,16,24}K\in\{2,4,8,16,24\} while keeping the encoder, predictor, training protocol, and planning budget fixed. We use the most challenging task from each benchmark: PushObj and Rotate.

TaskK=2K=2K=4K=4K=8K=8 (planar default)K=16K=16 (REALM-JEPA default)K=24K=24
PushObj (%)234586664
Rotate (%)0073.383.386.7

For the relatively simple PushObj scenes, K=8K=8 is already sufficient to achieve strong performance. Increasing KK allows the model to decompose objects into finer-grained slots and can provide further gains, but this comes with increased computation time. On Rotate, performance largely saturates around K=16K=16, with only a marginal improvement from further increasing KK. In contrast, when KK is too small (e.g., K=2K=2), the model can barely distinguish different objects and planning essentially fails. We therefore view the choice of K as a compute--performance trade-off: it should be selected according to the available computational budget, with a larger K preferred when resources permit.

Reviewer_VUDQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper essentially takes an action-conditioned V-JEPA model, adds a slot bottleneck, and predicts future spatial masks instead of patch features. It also adds a memory of previous transitions and changes the loss to measure spatial transport between masks. The results suggest this helps planning on the tested tasks. My main questions are whether the slot bottleneck will remain useful at scale, and how much of the improvement comes from the representation versus the new planning loss.

Critical strengths
  • The transport loss addresses a clear problem: pointwise feature distances can give poor guidance when an object is far from its goal.
  • Transition memory improves even the patch-based baseline, which is a useful result.
  • The planning results are strong, and the ablations support the proposed combination.
Critical weaknesses
  1. The paper is very hard to read. Many sentences feel empty, repetitive, or unnecessarily difficult to understand. The writing feels LLM-like, although I am not making a claim about how it was written. The core ideas are fairly straightforward, but the presentation makes them much harder to follow.

  2. The scalability of the slot bottleneck is unclear. Why restrict the representation if a larger patch-based model could learn the necessary structure? Once the decomposition is frozen, increasing predictor capacity cannot recover information it discarded. The current experiments show that this restriction helps on these tasks, but do not establish whether it remains helpful with more compute, data, and scene complexity.

  3. It is unclear how much of the gain requires slots. TMP already improves the patch-based baseline, while the transport loss provides a better spatial planning signal. Navigation also explicitly incorporates obstacle geometry. The results support the complete system, but the claims about object-centric modelling should be more carefully separated from improvements in memory and goal scoring.

Questions for authors

Questions

Does the advantage persist with a larger patch-based predictor or more training data? How much comes from better dynamics prediction versus better goal scoring?

Suggestions / Comments

  • Explain the method directly: what changes relative to an action-conditioned V-JEPA, what is predicted, and what is frozen.
  • State early that full training requires temporal transitions and that slots need not correspond to individual objects.
  • Add scaling experiments comparing patch and slot models across predictor sizes, training budgets, slot counts, and scene complexity. Report compute and memory alongside performance.
  • Substantially simplify the writing. Replace repeated abstract descriptions with concrete explanations.

Limitations

The paper acknowledges grouping and retrieval limitations. It should also discuss the fixed slot basis as a possible capacity restriction.

4 comments
AuthorsOpenReview ↗

Responses to W1 and W2

We sincerely thank the reviewer for recognizing the value of our work and for the thoughtful and constructive feedback! We have carefully revised the manuscript accordingly and provide our point-by-point responses below.

W1

The core ideas are fairly straightforward, but the presentation makes them much harder to follow.

Response to W1

We are sorry that our writing made the paper unnecessarily difficult to follow. Following your valuable suggestions in Q1, we will clarify the presentation in three ways:

  1. State the methodological changes directly. Starting from action-conditioned V-JEPA, we reorganize the encoder's patch representations into stable semantic slots and time-varying masks, which become the predictor inputs. We accordingly redesign the predictor and its context construction for slot--mask inputs, and replace V-JEPA's pointwise feature-ℓ1\ell_1 planning objective with a UOT-based mask objective.
  2. Introduce essential concepts earlier. We will define temporal transitions and the properties of our slots before using them to explain TMP and planning.
  3. Replace abstract descriptions with concrete examples. We will use representative planning cases to explain the correspondence problem, transition retrieval, and displacement-aware goal evaluation.

Please see our revision, especially Section 1 and Section 3.

W2

Why restrict the representation if a larger patch-based model could learn the necessary structure? Is it remain helpful with more compute, data, and scene complexity

Response to W2

This is a very interesting question and one that the representation-learning community has long explored: how information compression affects downstream tasks. From the information-bottleneck perspective, the goal of representation learning is to compress information, and this compression is irreversible. For example, V-JEPA compresses RGB information into a latent space and does not guarantee that textures, exact colors. The benefit is that downstream tasks become easier to learn: they can operate directly in the latent space without reconstructing the mapping from RGB to the latent space.

Similarly, for world models, we believe that patch-level latent representations can be further compressed into object-centric representations. At the cost of discarding some fine-grained information, this produces a transferable compression module that does not need to be reconstructed for each downstream task. Correspondingly, it reduces the learning difficulty of the predictor and makes the planning loss easier to design.

We divide the scalability discussion into two aspects:

1. Can scaling solve the problems of existing JEPA models?

Our slot decomposition and the corresponding TMP and UOT loss are not intended to provide a method that is easier to scale, but to explore problems that scaling alone cannot solve. Taking REALM-JEPA as an example, Table 4 shows that, because of the metric issue, even the planning loss produced by a ideal predictor remains significantly inconsistent with the true goal. Therefore, scaling up the predictor or increasing the amount of training data cannot significantly improve V-JEPA on this issue.

2. After the above problems have been solved, possibly through a method other than slots, will slots still retain a performance advantage under scaling?

Because these methods are all information bottlenecks, we can consider this question by referring to the necessity of pretrained representations. They are all forms of prior knowledge, and the advantages provided by such priors may indeed disappear with sufficiently large models and datasets. For example, after Kimi K3 was scaled to 2.8T parameters, it no longer required a pretrained visual encoder. Similarly, we believe that a sufficiently large predictor may eliminate the advantage brought by slots. However, we do not know where this boundary lies, and our current resources do not allow us to explore the scaling bound.

We update the above discussion to the limitation section.

AuthorsOpenReview ↗

Responses to W3 and Q1

W3

The results support the complete system, but the claims about object-centric modelling should be more carefully separated from improvements in memory and goal scoring.

Response to W3

It seems that our description did not make this experimental protocol sufficiently clear. TMP is not directly applied to a patch-based baseline. The “V-JEPA 2.1 + TMP” row only replaces the original V-JEPA predictor with the TMP architecture, but its contexts are still decomposed and retrieved using Slot-JEPA's slot--mask interface. Thus, this row evaluates the predictor change under our object-centric context construction, rather than a patch-based TMP that operates independently of slots. We will clarify this protocol and revise the row label to avoid this misunderstanding.

More broadly, slot decomposition, TMP, and UOT are structurally coupled rather than three independent additions. Slot decomposition first produces stable semantic components and temporally aligned masks. These representations enable TMP to retrieve transitions with consistent slot indices and UOT to transport mass between corresponding masks. Without slot decomposition, neither mask-transition memory nor slot-wise transport is applicable. Therefore, the effectiveness of TMP and UOT is built on the object-centric interface, and they complement object-centric modeling rather than provide slot-independent explanations for the gain.

Q1

Does the advantage persist with a larger patch-based predictor or more training data? How much comes from better dynamics prediction versus better goal scoring?

Response to Q1

As clarified in our response to W2, slots are intended to address problems that cannot be solved by scaling a patch-based predictor. For example, when planning with an ℓ1\ell_1 loss, even a perfect predictor that can exactly predict the representation after action execution still produces a planning loss that is inconsistent with the true goal. Therefore, a larger patch-based predictor or more training data cannot resolve this limitation of V-JEPA.

Our expanded ablation results show that both better dynamics prediction and better goal scoring are important:

VariantPlanar avg. (%)Reach (%)Rotate (%)Place (%)
V-JEPA 2.1--72.220.00.0
V-JEPA 2.1 + TMP--88.943.340.0
Slot-JEPA89.3100.083.376.7
w/o TMP84.744.46.6716.7
w/o UOT81.388.940.043.3
w/ Recent Context79.755.623.316.7
w/ Random Memory70.794.473.373.3

For example, replacing V-JEPA's predictor with TMP and retrieving contexts using slot masks effectively improves V-JEPA's performance. This part of the improvement may also be achievable with a larger predictor and more training data. However, based on practices in the VLM field, the amount of data or the number of parameters may need to increase by hundreds of times before a significant improvement appears. In contrast, better goal scoring cannot be achieved by scaling the predictor. It is also important to note that better dynamics prediction is the foundation for effective goal scoring, since even a well-designed planning loss cannot help if the predicted future state is inaccurate.

Reviewer_VUDQOpenReview ↗

Rebuttal acknowledgment

Thanks for the detailed response. The clarification that the V-JEPA + TMP variant still uses slot-mask contexts resolves my misunderstanding of that ablation. The added results strengthen the case for the combined system. I still see scaling the patch predictor under a matched goal-scoring objective as an open question. My rating remains 3 (weak accept).

AI assistance was used to draft this acknowledgment.

Thank you for your positive feedback and for acknowledging our additional results. We appreciate your constructive suggestions and will further explore the remaining open question in our future work. Thank you again for your time and consideration!

Reviewer_GXCWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper decomposes visual patch features into reference-derived slots and time-varying spatial masks. Its transition memory predictor (TMP) uses selected historical transitions to predict future masks, with separate branches for slot-specific changes and cross-slot interactions. Unbalanced optimal transport loss is used for prediction supervision and visual-goal evaluation. The paper evaluates visual understanding on seven benchmarks and planning in several robotic environments. I view its main contribution as the combination of the slot–mask interface and transition memory for visual-goal planning

I lean slightly toward weak acceptance: the TMP–slot integration and planning gains narrowly outweigh my concerns about attribution and representation scope. The results support its value for visual-goal planning, rather than establishing a generally superior world model. A convincing author response addressing these concerns could help me make a more informed rating decision.

Critical strengths
  1. I find the coupling between the representation and transition memory predictor (TMP) more compelling than any individual component. A shared slot basis provides consistent mask channels across observations, allowing historical visual transitions to be stored compactly and reused as prediction context. I consider this gives the structured representation a concrete role in memory and dynamics prediction, rather than using slots only for interpretability.

  2. Performance is good. On REALM-JEPA benchmark, the full method achieves 86.1% success, compared with 35.2% for V-JEPA 2.1 and 58.3% for V-JEPA 2.1 + TMP. Table 5 also compares different context-selection strategies while keeping TMP fixed, providing evidence that the choice of historical transitions matters beyond predictor capacity alone.

Critical weaknesses
  1. The contribution of the representation and objective is not fully isolated: Table 5 supports memory selection, but Table 1 does not separate slot decomposition from additional encoder fine-tuning (Appendix C.1). Similarly, removing UOT only compares it against mask l1l_1, leaving its advantage over other displacement-aware costs unclear. I consider UOT a reasonable design choice rather than a strong standalone contribution. These gaps limit the evidence for the proposed interface’s distinctive value beyond additional training and spatially informed goal evaluation.

  2. The applicability of the fixed-slot representation remains insufficiently characterized: With K=16K=16 and a fixed slot bank within each prediction context, Eq. (1) restricts reconstructed features to mixtures of reference-derived vectors. I believe the sensitivity to slot count and substantial visual-content changes remains unclear. This raises questions about when the compression remains adequate, rather than demonstrating a failure. I therefore find the evidence stronger for the evaluated visual-goal planning tasks than for a broadly reusable dynamics representation.

Questions for authors
  1. With the slot representation and TMP unchanged, have you compared UOT with another displacement-aware mask cost? Such a comparison would help distinguish the benefit of transport-based evaluation from the more general benefit of accounting for spatial displacement.

  2. I'm also curious about: What evidence supports the adequacy of K=16K=16 when scene content or object appearance changes substantially? Existing sensitivity results for KK, or failure cases associated with the slot bank before refreshing it, would help clarify the scope of the representation.

4 comments
AuthorsOpenReview ↗

Responses to W1

We sincerely thank the reviewer for recognizing the value of our work and for the thoughtful and constructive feedback! We have carefully revised the manuscript accordingly and provide our point-by-point responses below.

W1

Table 1 (visual-understanding gains) does not separate slot decomposition from additional encoder fine-tuning; UOT is a reasonable design choice rather than a strong standalone contribution.

Response to W1

Thank you for identifying these ambiguities. We would like to update an ablation that freezes all V-JEPA 2.1 encoder blocks and trains only the slot decomposition:

RepresentationCOCO mIoU (%)NYUv2 δ1\delta_1 (%)DAVIS J&F (%)
V-JEPA 2.152.880.565.9
Slot-JEPA, w/o encoder fine-tuning56.382.966.1
Slot-JEPA, with encoder fine-tuning56.183.166.6

The frozen-encoder variant still improves over V-JEPA 2.1, demonstrating that fine-tuning is not necessary for the gain. Moreover, Kinetics-400 is already included in V-JEPA 2.1 pretraining, so fine-tuning introduces no new dataset, and the encoder gradients come solely from the slot-decomposition objective. Fine-tuning therefore strengthens object awareness rather than explaining the gain through additional data.

We agree with the reviewer that the evidence does not support UOT's advantage over other displacement-aware costs. We would like to clarify that UOT is one natural implementation enabled by our slot--mask decomposition. It is not a standalone planning loss, but a component of the slot-based world model. Previous methods do not apply such a interface, thus typically use l1 or l2 losses, instead of displacement-aware costs. We will revise the corresponding claims to present UOT as our chosen displacement-aware implementation on top of the slot--mask decomposition rather than as a uniquely superior objective.

AuthorsOpenReview ↗

Responses to W2, Q1 and Q2

W2

The applicability of the fixed-slot representation remains insufficiently characterized.

Response to W2

It involves two separate issues: the capacity controlled by KK, and the scope of a fixed semantic slots.

1. Must KK remain fixed after pretraining?

No. KK does not need to match the value used during representation pretraining and can be adjusted according to the number of visually relevant components in each downstream task. We use K=8K=8 for planar planning and K=16K=16 for REALM-JEPA, and conducted a sensitivity study on the most challenging task from each benchmark while keeping the encoder, predictor, training protocol, and planning budget fixed:

TaskK=2K=2K=4K=4K=8K=8 (planar default)K=16K=16 (REALM-JEPA default)K=24K=24
PushObj (%)234586664
Rotate (%)0073.383.386.7

For the relatively simple PushObj scenes, K=8K=8 is already sufficient to achieve strong performance. Increasing KK allows the model to decompose objects into finer-grained slots and can provide further gains, but this comes with increased computation time. On Rotate, performance largely saturates around K=16K=16, with only a marginal improvement from further increasing KK. In contrast, when KK is too small (e.g., K=2K=2), the model can barely distinguish different objects and planning essentially fails. We therefore view the choice of K as a compute--performance trade-off: it should be selected according to the available computational budget, with a larger K preferred when resources permit.

2. Is the fixed-slot representation universally applicable and lossless?

We agree that it is not. Our slots are not intended as a general-purpose, lossless decomposition. They are specialized for world-model planning, trading some representational capacity for the temporal identity stability needed by planning. Updating complete slot vectors frame by frame creates a difficult correspondence problem when an object is occluded or leaves the view, especially from a wrist camera: the model must re-identify it and recover its previous slot when it reappears. Following Reviewer VeL6's suggestion (W4), we clarify how slot-jepa handle the following issues:

a) The following issues are specific to Slot-JEPA's fixed semantic slots:

  • Appearance changes. With a pretrained vision encoder, the same object can rotate, scale, articulate, or deform while retaining the same semantic vector in SS; MtM_t expresses how that object changes over time. This covers typical planning tasks, where an object does not abruptly become a semantically different object. Abrupt semantic changes remain outside the current scope.
  • State changes. Articulation or gripper opening alters the relative positions of object components and is therefore preserved through mask dynamics. Results on Reacher (hinge-angle changes), OGBench, and REALM-JEPA (gripper opening and closing) show that masks capture these states.

b) The following limitations are shared with image-based slot world models more broadly:

  • 3D pose. Like other image-based slot world models, Slot-JEPA does not encode an explicit object-level 3D pose. In REALM-JEPA, the projected masks from three views allow the predictor to infer 3D pose. In single-view 3D planning such as OGBench, like previous methods, Slot-JEPA relies on the fixed camera prior to infer 3D pose implicitly.
  • New objects during inference. The current slots cannot automatically allocate a slot to an object absent from the reference clip. Dynamic object birth is also largely unaddressed by existing slot-world-model benchmarks. We leave dynamic allocation to future work; periodically rebuilding the slot set is a practical intermediate solution.
  • Finite capacity and instance binding. A fixed number of slots is a common bottleneck. In this regime, however, Slot-JEPA's mask-based decomposition can be more informative since different instances assigned to the same semantic slot often appear as distinct spatial clusters in its mask, which a single slot vector does not guarantee.

Q1

Comparison with other displacement-aware mask cost.

Response to Q1

Please see the response to W1. We agree that the supported claim should be: a displacement-aware mask cost is more appropriate than pointwise ℓ1\ell_1 for spatial planning, rather than that UOT is superior to other displacement-aware mask costs. UOT is our implementation of this principle. We will revise the claim accordingly.

Q2

What evidence supports the adequacy of K=16K=16 when scene content or object appearance changes substantially?

Response to Q2

Please see the response to W2.

Reviewer_GXCWOpenReview ↗

Response to the Rebuttal

I appreciate the authors' revisions and responses. The added comparisons and ablations make the contribution of combining slots with TMP more convincing. The frozen-encoder results also help address my concern about attribution.

I still have some concerns about the method's generalization to more substantial scene changes, and I consider the overall novelty moderate. I therefore maintain my weak accept rating, but with stronger confidence in my judgement.

Thank you very much for your positive feedback and for acknowledging our efforts in addressing your concerns. We appreciate your constructive suggestions throughout the review process, which have helped us improve the paper.

Thank you again for your time and consideration!

Reviewer_azRGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper proposes Slot-JEPA, an object-structured framework for latent world models. It decomposes dense visual representations into stable semantic slots and time-varying spatial masks. The authors introduce a Transition Memory Predictor (TMP) to retrieve relevant historical transitions and use Unbalanced Optimal Transport (UOT) for future prediction and goal-conditioned planning. Experiments on visual understanding, planar navigation, and robotic manipulation demonstrate improvements over existing baselines.

Critical strengths

Clear motivation and reasonable framework design. The paper addresses the limitations of dense patch representations in world models. Combining structured visual representations, transition memory, and spatially aware planning is a meaningful direction.

Promising experimental results. Slot-JEPA achieves strong improvements on planar planning and robotic manipulation tasks, reaching 86.1% success on REALM-JEPA compared with 35.2% for V-JEPA 2.1. The paper also provides useful ablation studies and qualitative visualizations.

Critical weaknesses

Limited novelty and insufficient isolation of the main contributions. The framework mainly combines existing techniques, including Slot Attention, memory-based transition retrieval, and optimal transport. Although the paper provides ablation studies, it does not fully isolate the benefit of the proposed slot representation from the changes in prediction architecture and planning loss. In addition, the visual understanding improvements may partly result from additional encoder fine-tuning rather than slot decomposition itself.

Limited experimental evidence for generalization and practical applicability. The main manipulation evaluation contains only 108 simulated cases, and the component ablations do not cover the more challenging Rotate and Place tasks. The OOD planar experiments also initialize memory using transitions collected in the new test environments, making it difficult to assess generalization without such adaptation. Furthermore, the paper lacks planning latency and computational overhead analysis despite introducing additional memory retrieval and UOT operations.

Questions for authors

Could the authors provide more controlled comparisons that isolate the contributions of slot decomposition, TMP, and UOT, particularly on Rotate and Place tasks?

How does Slot-JEPA perform without test-environment memory initialization, and what is its end-to-end planning latency compared with the baselines?

5 comments
AuthorsOpenReview ↗

Responses to W1 (part 1)

We sincerely thank the reviewer for the detailed and constructive feedback! We have carefully revised the manuscript in response to these comments and provide our point-by-point responses below.

W1

Disscussion about novelty; the isolation of the slot representation from the changes in prediction architecture and planning loss; visual under standing improvements may partly result from encoder fine-tuning.

Response to W1

Thank you for raising these concerns! We address them separately below.

1. Does Slot-JEPA mainly combine existing techniques?

Thank you for highlighting that our presentation did not make the distinction from existing techniques sufficiently clear. We would like to clarify that our novelty is not a new Slot Attention operator or optimal-transport solver. Slot-JEPA aims to make a better object-centric dynamics prediction (please see the response to Reviewer VeL6 W1 for a comparison with more related routes). Sepcifically, we propose a new interface between patch representation and dynamic predictor with two key differences from prior object-centric dynamics models:

First, task-agnostic decomposition. We argue that organizing patch-level representations into objects is independent of the actions and objectives of a downstream task. Therefore, the slot module should precede the action-conditioned predictor as a transferable component. We pretrain Slot-JEPA on Kinetics-400, freeze it across all downstream tasks, and train only the task-specific dynamics predictor. In contrast, previous methods still treat slot extraction as part of downstream-task training, and finetune it on the target environment.

Second, decoupling object semantics from spatial dynamics. Given T frames, previous methods reorganize patch-feature into K×TK\times T slot vectors, while ours use KK shared vectros and K×TK\times T masks.

Its main advantage is temporal identity consistency: the same object remains tied to the same slot rather than being represented by different slots over time, avoiding re-localization after it leaves the view or becomes occluded and reducing the difficulty of long-horizon matching. This also enables slot–mask transition retrieval and displacement-aware mask-space planning. This design may be less suitable when object appearance changes abruptly, such as a sudden color change or the introduction of a new object. However, the visualizations in Fig. 5 show that, with strong pretrained representations, the shared semantic slots remains effective under rotation, scaling, deformation, and occlusion, which cover the common cases in planning tasks. Additionally, this slot definition prevents us from directly adopting DINOSAUR or SIMONe. We make three corresponding adaptations:

  • Initialization. Instead of DINOSAUR's randomly initialized single-stage Slot Attention, we extract local slots and merge them with farthest-point-initialized global Slot Attention.
  • Decoder. Instead of rendering per-slot features or RGB mixtures, patch features query the fixed slots to produce assignment masks.
  • Objective. Unlike DINOSAUR's static feature reconstruction or SIMONe's RGB ELBO, we optimize the decomposition as a feature reconstruction, slot orthogonality, and future-mask prediction.
AuthorsOpenReview ↗

Responses to W1 (part 2)

The following table provides a direct comparison with the closest object-centric dynamics methods. We group C-JEPA and SlotContrast-WM because SlotContrast-WM adopts the same OC-JEPA/C-JEPA predictor family.

AspectC-JEPA / SlotContrast-WMSlot-MPCSlotFormerSlot-JEPA
Slot decompositionVideoSAUR/SlotContrast trained on the target environmentSAVi trained on the target environmentSAVi/STEVE trained on the target environmentPretrained on generic videos and transferred without downstream retraining
Slot definitionK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK semantic slot vector and K*T dynamic spatial masks
Prediction contextContiguous observed slot historyContiguous observed slot historyContiguous burn-in slot historyRetrieved, non-contiguous transitions
Action-conditional predictionYesYesNoYes
Prediction lossMasked l2 loss on slotsl2 loss on slotsl2 loss on slotsUOT and l1 losses over masks
PlanningStandard CEM with l2 lossGradient-based MPC or MPPIRanks predefined PHYRE actionsStandard CEM with UOT and l1 losses over masks

2. Can the slot representation be isolated from TMP and UOT?

Perhaps due to our way of writting, it might leave you an impression that these components are independent. In fact, they are structurally coupled: stable slot semantics make masks comparable across time, enabling TMP to retrieve transitions with consistent slot indices and UOT to transport the mass of corresponding components. Without slot decomposition, neither mask-transition memory nor slot-wise transport is appliable. Removing slots while retaining the other two components is therefore not a meaningful ablation. Our ablations retain the slot interface and replace TMP with a sequential predictor or UOT with pointwise mask loss. On the 300 planar cases, success decreases from 89.3% to 84.7% and 81.3%, respectively, confirming the contribution of both components.

3. Do the visual-understanding gains merely come from encoder fine-tuning?

We add a control that freezes all V-JEPA 2.1 encoder blocks during representation pretraining and trains only the slot decomposition. The downstream encoder and decomposition remain frozen in all variants.

RepresentationCOCO mIoU (%)NYUv2 δ1\delta_1 (%)DAVIS J&F (%)
V-JEPA 2.152.880.565.9
Slot-JEPA, wo encoder fine-tuning56.382.966.1
Slot-JEPA, with encoder fine-tuning56.183.166.6

The frozen-encoder variant still improves over V-JEPA 2.1, demonstrating that fine-tuning is not necessary for the gain. Moreover, Kinetics-400 is already included in V-JEPA 2.1 pretraining, so fine-tuning introduces no new dataset. Besides, its encoder gradients come solely from the slot-decomposition objective. Fine-tuning therefore further strengthens object awareness rather than explaining the gain through additional data.

AuthorsOpenReview ↗

Responses to W2

W2

More manipulation results and ablations on the more challenging Rotate and Place tasks; OOD planar experiments without new transitions; planning latecy and computational overhead.

Response to W2

Thank you for raising these concerns. We address the three issues separately below.

1. More manipulation results and ablations. Following your valuable suggestion, we further extend the existing component ablations from Reach to the more challenging Rotate and Place tasks:

VariantPlanar avg. (%)Reach (%)Rotate (%)Place (%)
Slot-JEPA89.3100.083.376.7
w/o TMP84.744.46.6716.7
w/o UOT81.388.940.043.3
w/ Recent Context79.755.623.316.7
w/ Random Memory70.794.473.373.3

These experiments retain the same frozen encoder and slot decomposition, isolating the effects of predictor architecture, planning loss, and context selection under the shared slot interface.

2. OOD planar experiments without new transitions. TMP is designed for in-context adaptation: its parameters remain frozen at test time, and only a small number of transitions from the new environment are provided as context. We do not claim or aim to design a predictor which retains high OOD performance in a zero-shot setting without such context. Because the architecture requires context inputs, a strictly context-free evaluation is not defined. We instead replace all test-environment contexts with randomly sampled ID transitions from the training environments, without any OOD transition or parameter update.

MethodTwoRoomR (%)MazeR (%)PushObj (%)
DINO-CLS34702
ResNet12684
R3M8322
DINO-WM828628
V-JEPA 2.1508646
LeWM528830
Slot-JEPA, random ID context only407248
Slot-JEPA, test-environment context909658

Removing the contexts substantially reduces performance, especially under layout shifts in TwoRoomR and MazeR. This result is consistent with TMP's intended role: rapid in-context adaptation to new dynamics, rather than context-free zero-shot generalization. Other baselines might also benefit from transitions collected in the new environment, but these methods neither study nor implement such test-time contextual adaptation.

3. Planning latecy and computational overhead. We report world-model-side cost per MPC replanning step. All planar tasks use the same model architecture and search parameters. REALM-JEPA is reported separately because its planning settings differ.

BenchmarkMethodLatency / MPC step (s)Peak VRAM (GiB)FLOPs / MPC step (TFLOPs)
Planar (all tasks)LeWM0.130.580.08
Planar (all tasks)C-JEPA0.420.410.22
Planar (all tasks)V-JEPA 2.1 / DINO-WM1.26.410.3
Planar (all tasks)Slot-JEPA0.863.48.61
REALM-JEPAV-JEPA 2.1 / DINO-WM2.6611.225.1
REALM-JEPASlot-JEPA2.407.519.7

For Slot-JEPA, the additional retrieval takes 0.029 s per MPC step, and UOT takes 0.091 s. Consequently, Slot-JEPA still faster than V-JEPA and DINO-WM. C-JEPA and LeWM are efficient since they only feed several vectors into the predictor, but this might limit their capability.

DINO-WM does not provide a planner for multi-view robot manipulation. If its encoder weights are inserted into the same predictor architecture as V-JEPA 2.1, the cost are the same as VJEPA 2.1 with basically the same architecture. LeWM and C-JEPA currently cannot be transferred to REALM-JEPA's multi-view manipulation setting without designing a new planning interface.

Reviewer_VeL6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary:

The paper proposes Slot-JEPA: an object-structured interface between a pretrained JEPA encoder (V-JEPA 2.1 ViT-B/16) and a downstream action-conditioned predictor and planner. The dense feature representations are factorized into a fixed set of object slots via Slot Attention, and on time-varying masks.

Three proposed model has three key novel components:

  1. Slot decomposition module: pretrained on Kinetics-400 with feature reconstruction (L1) + slot orthogonality regularization.
  2. Transition Memory Predictor (TMP): stores transitions and retrieves 5 in-context transitions using the state and per-slot mask area/centroid statistics.
  3. Unbalanced OT (Sinkhorn-Knopp) between predicted and target masks of the same slot, used both as training loss and as planning cost. For navigation tasks the ground cost is "obstacle-aware", using the masks of the slots identified as static.

In the evaluations, the paper shows that Slot-JEPA outperfroms multiple baselines on dense visual tasks (segmentaiton, depth estimation, ...), and on visual planning.

Review:

The paper tackles a relevant problem: dense JEPA latents are an opaque interface for prediction and planning, and pointwise goal costs give poor guidance far from the goal. Decomposing the feature field into slots + masks, retrieving transitions as context, and using OT between masks as a goal cost is a reasonable and interpretable direction. The paper also contains a lot of work (new robotic benchmark, many evaluations, detailed appendix, released inference code).

However, the paper does not convincingly show that the object-structured interface is what makes the difference. Object-centric world models for planning already exist (e.g., SOLD, Slot-MPC, SlotFormer, Causal-JEPA), but none of them is used as a baseline or properly discussed, and most components are borrowed from prior work, which limits the novelty. The robotic evaluation relies on a self-made simulated benchmark, with apparently poorly tuned baselines and undefined success criteria. All experiments are run at a single, small scale (ViT-B/16, 10 epochs on Kinetics-400), so it is unclear whether the advantage holds at scale. The interface itself is only loosely object-centric (fixed slots, moving masks, a 2D image-space goal cost), and none of these design choices is ablated. Finally, the claims about dense interfaces that motivate the method are never tested. As a result, it is hard to tell how much of the improvement comes from the proposed interface, and how it compares to existing object-centric approaches.

I lean towards reject. I am willing to raise my score if the authors: (1) compare with slot-based world models (at least SOLD and Slot-MPC) or discuss them thoroughly, (2) show that the gains hold at a larger scale (at least a ViT-L encoder), (3) ablate the key design choices of the interface (fixed vs. evolving slots, mask vs. feature goal cost, number of slots), and (4) add experiments that test the motivating hypotheses (e.g., downstream data efficiency and memory cost against a dense interface).

Critical strengths
  • Well motivated. Dense patch latents are a redundant and opaque interface for planning, and pointwise L1/L2 goal costs are known to be flat far from the goal. Predicting masks and comparing them with transport is a natural and interpretable way to address this.

  • Interpretability: predicted future masks can be inspected (Fig. 13), and the loss landscapes (Fig. 2, 12) are a nice diagnostic.

  • The correlation analysis between goal loss and EE-goal distance (Tab. 4, C5) is a useful evaluation beyond success rates.

  • Retrieving non-contiguous transitions as in-context examples is an interesting idea, and the relevance/diversity ablation (Tab. 5, lines 3-5) supports it on PushT/PushObj.

  • Broad evaluation (7 probes, 6 planning environments, a new robotic benchmark), and large gains on some of these evals.

  • Inference code, configs, and evaluation endpoints are released for the planning tasks. The appendix is detailed.

Critical weaknesses
  1. Limited novelty and lack of discussion and comparison with slot-based world models. The central claim is that an object-structured interface improves prediction and planning, yet the paper does not compare against a single object-centric world model. Slot-based world models that plan or learn control from pixels already exist, and must be used as baselines or, at the very least, discussed. Some notable examples are:
    • SOLD (Mosbach et al., 2025): slot-based latent dynamics for model-based RL on relational manipulation from pixels.
    • Slot-MPC (Spieler et al., 2026): goal-conditioned MPC on top of slot-based world models for robotic manipulation, i.e., essentially the same setting as this paper on visual planning.
    • SlotFormer (Wu et al., 2023) and Causal-JEPA (Nam et al., 2026) are cited but not compared against, although Causal-JEPA is described as the closest counterpart.
    • Earlier object-centric world models for planning and control: OP3 (Veerapaneni et al., 2020), STOVE (Kossen et al., 2020), FOCUS (Ferraro et al., 2023), among others.

Beyond these, every component is borrowed: Slot Attention on SSL features (DINOSAUR), or SIMONe-style mask decoding. The related work in the main paper is a short paragraph, and the appendix version reads like a list, with many very relevant methods (see above) not mentioned.

  1. The robotic evaluation is weak. REALM-JEPA is the authors' own simulated benchmark. There are no real-robot experiments and no established benchmark. The only baselines are V-JEPA 2.1 variants, and V-JEPA 2.1 scores 0/30 on Place despite receiving ground-truth target-position guidance, which points to poorly tuned baselines. Success criteria are not defined, and it is unclear whether the baselines get the same inputs (multi-view CEM or context)

  2. Lack of experiments at scale: All experiments are run on the ViT-B scale. There is no proof that the proposed method will improve its performance when scaling pretraining data or compute. At least experiments at ViT-L size are needed to validate the claims at scale.

  3. The interface is not really object-centric, and its planning cost is a 2D image-space heuristic. In object-centric models (e.g., SAVi, SlotFormer, SOLD), slots carry the state of each object and evolve over time. Slot-JEPA inverts this: the slot vectors are fixed per environment and only their masks change. Visual information that is not mask geometry is lost: appearance or state changes (e.g., articulated objects, gripper opening), 3D pose, objects outside the reference set, or more entities than slots. Slots are not bound to objects either (as the authors acknowledge), so instances sharing a slot cannot be told apart. Likewise, the UOT goal cost measures mass displacement in the image plane of each camera, which is a reasonable proxy for the top-down 2D tasks but not a principled distance in 3D. None of these design choices is justified or ablated, e.g., fixed vs. evolving slot vectors, a goal cost on masks vs. on reconstructed features, or the number of slots K.

  4. The motivation is never tested. The paper motivates Slot-JEPA with three limitations of dense interfaces: predictors must redundantly recover object structure, patch history is expensive to store and query, and pointwise costs are insensitive to displacement. However, the first two are never measured. I encourage the authors to design experiments that validate these hypotheses, e.g., downstream data efficiency against a dense interface, transfer across tasks, and the storage/compute cost of mask vs. dense memor

Questions for authors
  • Can you run experiments with a larger encoder size to validate the claims at scale?

  • Does the proposed method extend beyond the V-JEPA 2.1 encoder? For instance, does it tranfer to other SSL methods (DINOv3, CAPI, ...) and scales?

  • What is the planning time per MPC step (planar and robot) compared to DINO-WM / LeWM / V-JEPA 2.1?

  • Regarding the REALM-JEPA benchmark: what are the success criteria for each task family? Do the baselines get the same warm-up, multi-view CEM, and memory? Why not compare with the released V-JEPA 2-AC? Any plans for real-robot or established-benchmark experiments?

  • How many slots are used for the planning tasks and for REALM-JEPA? How sensitive are the results to the predefined number of slots?

7 comments
AuthorsOpenReview ↗

Responses to W1 (part 1)

We sincerely thank the reviewer for the detailed and constructive feedback! We have carefully revised the manuscript in response to these comments and provide our point-by-point responses below.

W1

Discussion and comparison with existing slot-based world models to clarify the novelty.

Response to W1

Thank you for bringing these relevant works to our attention! We structure our response in two parts. First, we classify the role of these slot-based methods. We then explain Slot-JEPA's novelty at three levels: its objective, its methodology, and its implementation.

Because both “object-centric” and “world model” are broad concepts, these methods appear to address a similar topic but in fact focus on distinct questions and stages of the pipeline. They broadly follow the same pipeline: visual observations are converted into slots, a dynamics predictor models how the slots evolve, and the learned predictor is used either for online planning or for policy learning. We summarize them as follows:

DirectionCore questionRepresentative methodsRelation to Slot-JEPA
Object-Centric Representation LearningHow are visual observations converted into slots?DINOSAUR, SIMONe, SAVi, SlotContrastRelated representation-learning precedents. Slot-JEPA trains its own decomposition module on V-JEPA 2.1 features
Object-Centric Dynamics PredictionHow does the object-centric state evolve over time and, when available, under actions?C-JEPA, SlotContrast-WM, SlotFormer, Slot-MPCSlot-JEPA belongs to this category
Object-Centric Model-Based PlanningHow is the learned model used to select actions online?OP3, STOVEUses predicted rollouts for direct planning
Object-Centric Model-Based RLHow is the learned model used to train policies?FOCUS, SOLDUses object-centric rollouts for actor/critic learning

Compared with other object-centric dynamics prediction methods, Slot-JEPA has the following differences:

1. Task-agnostic decomposition. We argue that organizing patch-level representations into objects is independent of the actions and objectives of a downstream task. Therefore, the slot module should precede the action-conditioned predictor as a transferable component. We pretrain Slot-JEPA on Kinetics-400, freeze it across all downstream tasks, and train only the task-specific dynamics predictor. In contrast, previous methods still treat slot extraction as part of downstream-task training, and finetune it on the target environment.

2. Decoupling object semantics from spatial dynamics. Given T frames, previous methods reorganize patch-feature into K×TK\times T slot vectors, while ours use KK shared vectros and K×TK\times T masks. C-JEPA, SlotContrast-WM, Slot-MPC, and SlotFormer all follow the SAVi-style formulation: they map Zt∈RN×DZ_t\in\mathbb{R}^{N\times D} to a complete slot state St∈RK×DS_t\in\mathbb{R}^{K\times D} and predict the evolution of the slot vectors.

Slot-JEPA instead extracts a shared semantic vectors S∈RK×DS\in\mathbb{R}^{K\times D} from reference features ZR∈RTR×N×DZ_{\mathcal R}\in\mathbb{R}^{T_{\mathcal R}\times N\times D} and represents each frame by a spatial assignment Mt∈[0,1]K×NM_t\in[0,1]^{K\times N}. Its main advantage is temporal identity consistency: the same object remains tied to the same slot rather than being represented by different slots over time, avoiding re-localization after it leaves the view or becomes occluded and reducing the difficulty of long-horizon matching. This also enables slot–mask transition retrieval and displacement-aware mask-space planning. This design may be less suitable when object appearance changes abruptly, such as a sudden color change or the introduction of a new object. However, the visualizations in Fig. 5 show that the shared semantic slots remains effective under rotation, scaling, deformation, and occlusion, which cover the common cases in planning tasks.

AuthorsOpenReview ↗

Responses to W1 (part 2)

The following table provides a direct comparison with the closest object-centric dynamics methods. We group C-JEPA and SlotContrast-WM because SlotContrast-WM adopts the same OC-JEPA/C-JEPA predictor family.

AspectC-JEPA / SlotContrast-WMSlot-MPCSlotFormerSlot-JEPA
Slot decompositionVideoSAUR/SlotContrast trained on the target environmentSAVi trained on the target environmentSAVi/STEVE trained on the target environmentPretrained on generic videos and transferred without downstream retraining
Slot definitionK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK*T time-varying slot vectors encoding both semantics and dynamicsK semantic slot vector and K*T dynamic spatial masks
Prediction contextContiguous observed slot historyContiguous observed slot historyContiguous burn-in slot historyRetrieved, non-contiguous transitions
Action-conditional predictionYesYesNoYes
Prediction lossMasked l2 loss on slotsl2 loss on slotsl2 loss on slotsUOT and l1 losses over masks
PlanningStandard CEM with l2 lossGradient-based MPC or MPPIRanks predefined PHYRE actionsStandard CEM with UOT and l1 losses over masks

3. Implementation. This slot definition prevents us from directly adopting DINOSAUR or SIMONe. We make three corresponding adaptations:

  • Initialization. Instead of DINOSAUR's randomly initialized single-stage Slot Attention, we extract local slots from spatiotemporal V-JEPA features and merge them with farthest-point-initialized global Slot Attention. The resulting ordered bank is fixed across current, predicted, and goal states.
  • Decoder. Instead of rendering per-slot features or RGB mixtures, patch features query the fixed bank to produce assignment masks, with z~t,n=∑kMt,k,nsk\widetilde z_{t,n}=\sum_k M_{t,k,n}s_k.
  • Objective. Unlike DINOSAUR's static feature reconstruction or SIMONe's RGB ELBO, we optimize the decomposition as a dynamics interface: feature reconstruction preserves the V-JEPA space, slot orthogonality prevents a redundant fixed basis, and future frames decoded against the same bank provide mask targets for joint future-mask prediction.

Because the primary goal of this work is not to design a better general-purpose slot extractor, we omitted these implementation distinctions. We apologize that this leads to a misunderstanding that the module directly assembles existing components, and we will describe its architecture, objectives, and differences from DINOSAUR and SIMONe in the revision.

AuthorsOpenReview ↗

Responses to W2

W2

Robotic evaluation on established benchmark; low performace of V-JEPA 2.1 on the Place task; the same success criteria and inputs for baselines.

Response to W2

We agree that evaluation on an established benchmark is necessary, and update another robotic benchmark OGBench including the two closest object-centric world models (C-JEPA and SlotContrast-WM). Slot-MPC is also applicable to this benchmark in principle. However, its data, code, and checkpoints are still listed as “coming soon”, so a reproducible comparison is currently infeasible. We will add it once these resources are released. By the way, we also update another benchmark Reacher suggested by Reviewer 9NK8:

MethodOGBench success (%)Reacher success (%)
DINO-CLS6884
ResNet6288
R3M6888
DINO-WM8482
V-JEPA 2.18088
LeWM7486
C-JEPA7282
SlotContrast-WM7070
Slot-JEPA9498

OGBench provides a standardized comparison, but its visual planning environments have limited background and object diversity, a single viewpoint, and relatively simple tasks. Therefore we use REALM-JEPA as a complementary stress test to validate whether a pretrained slot module transfers across visual changes, viewpoints, occlusion, and multi-stage manipulation.

To diagnose V-JEPA 2.1's Place result, we provide a stage-wise diagnosis of the remaining V-JEPA 2.1 failures below:

Failed stageCasesProportion
Approach object5/3016.7%
Grasp object24/3080%
Move/place object1/303.3%

Most failures (80%) occur during grasping, which requires precise gripper control and is therefore most sensitive to prediction errors. This is consistent with the conclusion in V-JEPA 2's paper: grasping requires precise control from visual feedback, and inaccurate actions or gripper opening cause failures. Crucially, when we keep all settings (representation, three-view inputs, visual goals, action space, CEM budget, controller) unchanged, and replace only the sequential predictor with TMP, Place success rises from 0% to 40%. This controlled comparison argues against poor tuning as the cause of the low V-JEPA 2.1 result. Instead, it shows the importance of memory for prediction in precision-critical manipulation.

All methods perform planning from the same synchronized three-view observations (two fixed external cameras and one wrist camera) and use the same visual goals, action space, CEM budget, and success criteria (detailed as follows):

TaskStrict success criterion
ReachThe target object contacts the gripper.
PickAfter reaching and grasping, the object's height increases by more than 7.5 cm.
RotateAfter reaching and grasping, the object's rotation differs from its initial orientation by more than 1.1 rad.
PlaceThe object is released inside the target container (or on top of a tray-like receptacle).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Major Responses and Revisions

Dear Reviewers,

Thank you very much for your valuable feedback and constructive suggestions! To save your time and facilitate the discussion, we have summarized our key responses to your concerns below. We have also uploaded a revised manuscript incorporating the corresponding changes.

  1. Novelty and positioning. We clarify that Slot-JEPA introduces a task-agnostic slot decomposition and decouples object semantics from spatial dynamics using fixed semantic slots and dynamic masks. This preserves temporal identity and enables mask-transition retrieval and displacement-aware planning. We also add comparisons with C-JEPA, SlotContrast-WM, Slot-MPC, and SlotFormer. (Reviewer VeL6 W1; Reviewer 9NK8 W1; Reviewer azRG W1)

  2. Representation scope and controlled analysis. A frozen-encoder ablation confirms that the visual gains come from slot decomposition rather than additional encoder fine-tuning. We also clarify the limitations of fixed slots and add a sensitivity study to KK. (Reviewer GXCW W1–W2, Q2; Reviewer azRG W1; Reviewer VeL6 W4, Q5; Reviewer 9NK8 Q1)

  3. Expanded benchmarks and stronger baselines. We add OGBench and Reacher, where Slot-JEPA achieves 94% and 98%, as well as comparisons with C-JEPA, SlotContrast-WM, and RoboJEPA-8B. We also clarify that Slot-JEPA’s advantage remains substantial in the challenging vision-only and OOD settings. (Reviewer VeL6 W2, Q4; Reviewer 9NK8 W1–W4)

  4. Stronger ablations and OOD analysis. New Rotate and Place ablations show that both TMP for dynamics prediction and UOT for goal scoring contribute substantially. Experiments without test-environment transitions further confirm that TMP provides in-context adaptation, rather than context-free zero-shot generalization. (Reviewer VUDQ W3, Q1; Reviewer azRG W1–W2, Q1–Q2; Reviewer VeL6 W4–W5)

  5. Scaling and encoder transfer. Experiments with V-JEPA 2.1 ViT-L and DINOv3 ViT-B show that Slot-JEPA scales to larger encoders and transfers across self-supervised representation families. We also clarify that scaling can improve prediction, but cannot by itself resolve an unsuitable goal-scoring metric. (Reviewer VeL6 W3, Q1–Q2; Reviewer VUDQ W2, Q1)

  6. Efficiency, evaluation protocol, and presentation. We report latency, VRAM, and FLOPs, showing that Slot-JEPA is more efficient than V-JEPA 2.1/DINO-WM and reduces transition storage by 48×. We also clarify the shared evaluation protocol and success criteria, moderate the claims about UOT, and simplify the method presentation throughout the paper. (Reviewer VeL6 W2, W5, Q3–Q4; Reviewer azRG W2, Q2; Reviewer GXCW W1, Q1; Reviewer VUDQ W1)

We would greatly appreciate it if you could review our responses and revisions and let us know whether any concerns remain. If you find that your concerns have been adequately addressed, we would be grateful if you could consider raising your rating accordingly.

Thank you again for your time and consideration!

Best regards,

The authors

AuthorsOpenReview ↗

Follow-up on Rebuttal and Remaining Concerns

Dear Reviewers,

Thank you again for your valuable feedback and constructive suggestions.

As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and revised manuscript and let us know whether any concerns remain. If you feel that your concerns have been adequately addressed, we would be grateful if you could consider updating your ratings accordingly.

Thank you for your time and consideration!

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.