acceptodds
Under review as a conference paper at ICLR 2027

Entity-Centric Exploration through Latent Dynamics for Visual Reinforcement Learning

Abstract

Exploration is central to reinforcement learning, but it remains challenging in long-horizon tasks with high-dimensional visual inputs. In many such tasks, important structure is not captured at the level of raw pixels or full states, but at the level of entities and their interactions. In this work, we explore how entity structure can be leveraged to make exploration more efficient under visual observations. Through a theoretical connection between dynamics-prediction error and count-based exploration via information gain, we show that exploiting entity structure can reduce exploration complexity, which motivates our entity-centric approach to exploration. Inspired by this, we introduce ECLAIR, an entity-centric exploration method that learns unsupervised, entity-centric representations online and uses a latent dynamics model to estimate entity-level novelty. By training online, ECLAIR allows the entity abstraction to improve as the agent encounters new states. We evaluate ECLAIR on challenging sparse-reward visual RL tasks and demonstrate that its exploration efficiency outperforms prior exploration baselines across skill-based navigation, dynamic control, and robotic manipulation. Additional visualizations and video rollouts are available on our anonymous website: https://sites.google.com/view/eclair-rl.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Reinforcement Learning, Object Centric Learning, Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UhVrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ECLAIR, an entity-centric exploration framework for sparse-reward visual RL. It combines unsupervised object-centric representations with latent dynamics prediction to construct entity-level intrinsic rewards. The paper also connects prediction errors to pseudo-count-based exploration and analyzes the benefits of factored reward structures. Experiments across MiniHack, SpriteWorld, and OGBench show promising improvements over established baselines.

Critical strengths
  1. Effective integration of entity-centric representations and latent dynamics. ECLAIR combines object-centric representations with latent dynamics to provide structured exploration signals. Its online training mechanism allows representations and dynamics models to adapt to newly encountered states, extending approaches that rely on fixed pretrained representations.

  2. Strong empirical results and useful analyses. ECLAIR achieves promising results across diverse environments, particularly on challenging long-horizon tasks. The ablations support the importance of online fine-tuning and KL clipping, while the visualizations suggest that intrinsic rewards align with meaningful object interactions.

Critical weaknesses
  1. The contribution of entity-centric factorization could be better isolated. Although patch-based and slot-based variants are evaluated, differences in model architectures and experimental settings make it difficult to isolate the benefits of entity-centric factorization from those of the underlying dynamics model. More controlled comparisons would strengthen the central claim.

  2. Limited cross-domain validation of key design choices. While the paper evaluates alternative representations across several tasks, the main component-level ablations, including online fine-tuning and prediction horizon, are primarily conducted on Scene-T1. It remains unclear whether these findings generalize consistently across different domains. Extending these ablations to other environments would strengthen the conclusions.

Questions for authors
  1. Relationship to prior entity-centric RL methods. Could the authors clarify how ECLAIR differs from ECRL (2024) and HECRL (2026), particularly given that HECRL also addresses sparse-reward manipulation? A brief discussion emphasizing ECLAIR's focus on online exploration would help clarify its contribution.

  2. Relationship to JEPA-style predictive models. How does ECLAIR relate to JEPA-style latent predictive models? Could similar exploration signals be obtained from a JEPA-style predictor, and what specific advantages does entity-centric factorization provide over monolithic latent prediction?

3 comments

We thank the reviewer for their positive assessment of the paper and their constructive feedback. We address each concern below.

Isolating the benefit of entity-centric factorization.

We clarify that the patch variant is the closest controlled comparison. Architecturally, it changes only the encoder, from object-centric particles to a fixed grid of image patches. The dynamics module, the number of latents, and the latent dimension are the same as in ECLAIR. The prediction horizon HH is tuned separately for each variant over {1,5,10}\{1, 5, 10\}, and we report the best setting. For patches, H = 1 performed best, so the reported result reflects its strongest configuration, at a higher compute cost. Similarly, for every method, we search the intrinsic reward scale β\beta over {0.1,1.0,2.0}\{0.1, 1.0, 2.0\} per domain and report the value with the best performance, which is β=0.1\beta = 0.1 for patches. Under this comparison, the entity-centric encoder improves the reward from 0.89 to 0.99 on MultiRoom-N4 and from 0.78 to 0.89 on MultiRoom-N6 (Table 2).

For slots, we made a substantial effort to incorporate a second entity-centric approach into the framework. No latent world model for slots comparable to LPWM exists, so we used the existing SOLD implementation (Mosbach et al., 2025), which pairs a SAVi encoder (Kipf et al., 2022) with a deterministic slot dynamics model. Its bonus is therefore the prediction error (MSE) in slot space. Under equal, fixed isotropic variance, the KL divergence between two Gaussians reduces to a scaled squared error, so slot-space MSE is the fixed-variance special case of our latent-KL bonus rather than a different objective.

The goal of these variants is to show that the framework applies across entity-centric representations, not to rank the representations themselves. We have clarified this in Appendix C.1.

Ablations beyond Scene-T1.

Following the reviewer's suggestion, we extended the ablations to MultiRoom-N4, MultiRoom-N6, and Scene-T2 (Table R2 in the general response; Appendix G). The main conclusions generalize across domains. Online fine-tuning matters in every domain. KL clipping is important in the Scene manipulation tasks and neutral in MultiRoom. ECLAIR is robust to the prediction horizon on MultiRoom and Scene-T1, while a longer horizon helps on Scene-T2. We thank the reviewer for suggesting these experiments, which we believe strengthen the paper.

AuthorsOpenReview ↗

Author's response (continued)

Relationship to ECRL and HECRL.

ECRL and HECRL use DLP-style entity representations inside the policy and value function for goal-conditioned manipulation. ECLAIR addresses a different problem, which is exploration in sparse-reward tasks without goals or demonstrations, and uses entity structure in the intrinsic reward.

The second key difference is that ECLAIR fine-tunes its entity representation online. An exploring agent keeps reaching configurations that its initial data does not cover, so the representation must adapt online. This is a major challenge that ECRL and HECRL do not address, as both prior works learn a frozen representation from a fixed offline dataset and never fine-tune it online. Our ablations confirm that this adaptation matters in every domain (Table R2).

Finally, the two lines of work are complementary. We can leverage ECLAIR to provide an exploration signal for ECRL/HECRL-style learners. We will leave this for future work. We have added this comparison to Appendix B.

Relationship to JEPA-style predictors.

JEPA-style models (LeCun, 2022; Assran et al., 2023) predict in a learned latent space without reconstruction. ECLAIR's entity model is trained with reconstruction, which anchors each entity latent to objects in the image better. When learning online from a changing data distribution, this mitigates the problem of latent space drifting or collapsing. The closest object-centric JEPA we are aware of is C-JEPA (Nam et al., 2026), which we found difficult to adapt to our online exploration setting. We leave the incorporation of JEPA-related world models to future work and have added JEPA and C-JEPA to Appendix B.

We hope to have addressed all the concerns from the reviewer, and we welcome any further comments.

References

  • Assran, M., et al. (2023). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR.
  • Daniel, T., & Tamar, A. (2022). Unsupervised Image Representation Learning with Deep Latent Particles. ICML.
  • Haramati, D., Daniel, T., & Tamar, A. (2024). Entity-Centric Reinforcement Learning for Object Manipulation from Pixels. ICLR.
  • Haramati, D., et al. (2026). Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion. ICLR.
  • Kipf, T., et al. (2022). Conditional Object-Centric Learning from Video. ICLR.
  • LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. OpenReview preprint.
  • Mosbach, M., et al. (2025). SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels. ICML.
  • Nam, H., et al. (2026). Causal-JEPA: Learning World Models through Object-Level Latent Masking. ICML.

Thanks to the authors for the response.

The authors have addressed most of my concerns. I will keep my positive score.

Reviewer_ymWzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ECLAIR proposes an exploration method for sparse-reward visual RL tasks. It learns entity-centric representations online and uses a latent dynamics model to estimate entity-level novelty. By training online, ECLAIR allows the entity abstraction to improve as the agent encounters new states. Experiments demonstrate that ECLAIR achieves greater exploration efficiency than prior exploration baselines across skill-based navigation, dynamic control, and robotic manipulation.

Critical strengths
  1. ECLAIR uses per-entity dynamics prediction error as a structured estimator for pseudo-counts over the entity state space, thereby facilitating entity-centric exploration. This exploration highlights object interactions, such as opening doors or pulling drawers open, thereby encouraging further physical interactions. ECLAIR discovers entities in an unsupervised manner. This method allows the representation to improve as the agent collects more informative experiences.

  2. The experiments cover multiple benchmarks and tasks, providing a relatively comprehensive and reliable evaluation. Experiments of model variants, and ablation studies further confirm that ECLAIR’s performance improvement stems from the proposed exploration mechanism.

Critical weaknesses
  1. The theoretical analysis assumes that task rewards can be decomposed across entities and that these entity-specific rewards can be queried independently. In practice, ECLAIR receives only environment-level reward feedback and cannot query these individual entity rewards. Furthermore, the current derivation of Proposition 1 considers only a fixed-variance mean-update approximation, and Appendix A.1 assumes that the dynamics variance remains fixed. However, the practical method jointly trains the entity encoder and dynamics model online, changing the predictive mean, variance, and latent representation space simultaneously. Therefore, the extent to which these theoretical analyses support the practical implementation requires further clarification.

  2. Compared with the main model, the patch variant not only uses fixed image patches as representations but also switches from H-step open-loop surprise to one-step surprise and changes the intrinsic reward weight. The slot variant changes both the encoder and the dynamics model, replacing latent KL with slot-space prediction error. Because these comparisons change multiple factors simultaneously, it is difficult to determine how much of the performance improvement is attributable to the entity representation.

Questions for authors
  1. Since entity-specific rewards cannot be queried independently in the actual implementation, how the theoretical analysis supports ECLAIR’s practical implementation requires clarification. Furthermore, how the fixed-variance mean-update approximation in Proposition 1 is consistent with the online joint training of the encoder and dynamics model also requires clarification, because the predictive mean, variance, and latent representation space all change during joint training. Under what conditions can the theoretical analysis still support ECLAIR’s practical implementation?

  2. Are there controlled comparisons that vary only the entity representation while keeping all other factors unchanged? Only such controlled comparisons would allow us to reliably attribute performance differences to the entity representation. The current patch variant switches from H-step open-loop surprise to one-step surprise, while the slot variant switches from latent KL divergence to slot-space prediction error. Thus, these variants change more than just the entity representation.

1 comment

We thank the reviewer for their positive assessment of the paper and their constructive feedback. We address each concern below.

How the theory supports the practical method.

We want to clarify that the theory motivates the design of ECLAIR. It is not meant as a direct description of the practical algorithm.

Per-entity rewards. We agree that the bandit analysis assumes an additive reward whose per-entity components can be queried separately. This is an idealization that we use to show how much factoring can reduce exploration complexity, and ECLAIR does not rely on it. In practice, the analysis supports the practical method when novelty is estimated per entity, which holds by construction, although the task reward is used as a single environment-level signal.

Fixed variance. Proposition 1 uses simplified Gaussian assumptions with a fixed dynamics variance to motivate the KL bonus as a pseudo-count proxy. The approximation holds when the encoder and the predictive variance change slowly relative to the per-entity mean predictions. In practice, warm-starting the model, a small online learning rate, and per-particle KL clipping keep this condition approximately satisfied.

We state these conditions explicitly and discuss when the analysis applies in the new Appendix A.4.

Controlled comparison of entity representations.

We clarify that the patch variant is the closest controlled comparison. Architecturally, it changes only the encoder from object-centric particles to a fixed grid of image patches. The dynamics module, the number of latents, and the latent dimension are the same as in ECLAIR. The prediction horizon HH is tuned separately for each variant over {1,5,10}\{1, 5, 10\}, and we report the best setting. For patches, H = 1 performed best, so the reported result reflects its strongest configuration, at a higher compute cost. Similarly, for every method, we search the intrinsic reward scale β\beta over {0.1,1.0,2.0}\{0.1, 1.0, 2.0\} per domain and report the value with the best performance, which is β=0.1\beta = 0.1 for patches. Under this comparison, the entity-centric encoder improves the reward from 0.89 to 0.99 on MultiRoom-N4 and from 0.78 to 0.89 on MultiRoom-N6 (Table 2).

For slots, we made a substantial effort to incorporate a second entity-centric approach into the framework. No latent world model for slots comparable to LPWM exists, so we used the existing SOLD implementation (Mosbach et al., 2025), which pairs a SAVi encoder (Kipf et al., 2022) with a deterministic slot dynamics model. Its bonus is therefore the prediction error (MSE) in slot space. Under equal, fixed isotropic variance, the KL divergence between two Gaussians reduces to a scaled squared error, so slot-space MSE is the fixed-variance special case of our latent-KL bonus rather than a different objective.

The goal of these variants is to show that the framework applies across entity-centric representations, not to rank the representations themselves. We have clarified this in Appendix C.1.

We hope to have addressed all the concerns from the reviewer, and we welcome any further comments.

References

  • Kipf, T., Elsayed, G. F., Mahendran, A., Stone, A., Sabour, S., Heigold, G., Jonschkowski, R., Dosovitskiy, A., & Greff, K. (2022). Conditional Object-Centric Learning from Video. ICLR.
  • Mosbach, M., Ewertz, J. N., Villar-Corrales, A., & Behnke, S. (2025). SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels. ICML.
Reviewer_xcpcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of object-level exploration, where effective exploration is aligned with the agent's interactions with objects in the environment and cross-interactions between objects. The paper proposes an object- or entity-centric exploration objective, which is based on prediction error over a latent dynamics model, learned at the top of entity-centric representations. The paper also shows the relationship between the prediction error and pseudo-counts, and shows that their objective reduces exploration complexity in a simplified bandit setting. The method is compared to several common exploration baselines in various environments with both discrete and continuous control, the ablation study provides some analysis of which components of the algorithm are important for performance.

Critical strengths
  • The paper's motivation is clear, and the results are strong.
  • The results show that the method works well in both discrete and continuous control environments.
  • The paper provides theoretical justification for its approach in terms of exploration complexity.
Critical weaknesses
  • The performance seems to depend somewhat on the type of model used for representation learning and dynamic modelling, although the ablation shows that the method's performance is still competitive.
  • Computing the intrinsic reward every H steps can be limiting in some environments; it also shows that computing the intrinsic reward is expensive at each time step.
  • It seems there is an implicit assumption about the temporal closeness between different object interactions; for example, if there is a large temporal delay between holding a key and finding the door to unlock it, the method might struggle. Can the author conduct an experiment to investigate this possible limitation?
  • I find some of the environments (KeyRoom and MultiRoom) used for experimentation to be a bit simple and do not need a long horizon for interacting with objects; could perform some experiments on, for example, MiniGrid door-key environment(s)
Questions for authors
  • I wonder how the method will work in a Minigrid doorkey environment(s)
2 comments

We thank the reviewer for their positive assessment of the paper and their constructive feedback. We address each concern below.

Dependence on the representation and dynamics model.

We agree that the choice of model matters, and we see this as a strength of our method, which is a framework that can be built on multiple representations and dynamics models, as we show in our experiments. ECLAIR only requires an entity-centric representation and a latent dynamics model over it, and all three instantiations we tested (particles, patches, and slots) remain competitive, as the reviewer notes. Our core contribution is the entity-centric exploration objective and its connection to count-based exploration, rather than a particular representation or dynamics model.

Cost of computing the reward every HH steps.

We see HH as a compute/efficiency trade-off: a smaller HH is more expensive but can capture full object interactions when they happen and assign novelty immediately, while a larger HH has lower cost but has less fine-grained reward. Our new ablations (Table R2) show that ECLAIR is robust to HH on most tasks and benefits from a longer HH on Scene-T2, so the longer HH is a good default. We have made this trade-off explicit in Section 4.3 and Appendix G of the revised paper. Despite the challenge on compute, we also conduct compute-matched experiments in Appendix G (Table 8), which show that ECLAIR still outperforms the baselines under a matched compute budget.

Long delays between object interactions / MiniGrid DoorKey.

ECLAIR does not assume that interactions are close in time. Without entity structure, ECLAIR reduces to the latent-dynamics bonus of prior work, such as our Dynamics baseline (Stadie et al., 2015; Hafner et al., 2023), so it should perform comparably to these methods. Entity structure improves on this, and every variant we tested matches or outperforms the Dynamics baseline.

We evaluated ECLAIR on MiniGrid DoorKey, where the delay between picking up the key and unlocking the door grows with the grid size (Table R1; Appendix F). On the smaller grids, even PPO without an exploration bonus solves the task, while it largely fails on our MiniHack KeyRoom with the same interactions, which shows that the MiniHack versions are harder to learn from pixels. On the largest grid, all baselines fail, and ECLAIR is the only method with non-trivial success. This shows that ECLAIR's exploration signal remains useful even when object interactions are far apart in time.

We thank the reviewer for suggesting these experiments, which we believe strengthen the paper. We welcome any further comments.

References

  • Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering Diverse Domains through World Models. arXiv:2301.04104.
  • Stadie, B. C., Levine, S., & Abbeel, P. (2015). Incentivizing Exploration in Reinforcement Learning with Deep Predictive Models. arXiv:1507.00814.

Thanks to the authors for the response.

The authors have answered most of my concerns. However, I will keep my score as I do not think this work is a breakthrough, but I still think it is a good contribution.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all reviewers for their constructive feedback. Reviewers found the motivation clear (xcpc, UhVr), the results strong across discrete and continuous control (xcpc, ymWz, UhVr), and the integration of entity-centric representations with latent dynamics effective (UhVr). We address each concern in the individual responses, and we summarize new experiments conducted to address each reviewer's concerns below.

We include new experiments on the MiniGrid DoorKey suite in Table R1 (per xcpc) to evaluate our method on tasks with sparse object interactions. We also extend our component ablations to three more tasks in Table R2 (per UhVr). We have also revised the paper to incorporate the new results in blue (in Appendix F and G). Overall, these results show that ECLAIR's performance remains competitive as the object interactions become less frequent in a task, and that our ablation results generalize across different domains.

Table R1. Success rate on MiniGrid DoorKey from pixels (mean ± s.e. across 4 seeds). Bold indicates best performing method(s).

Method5x56x68x816x16
PPO1.00 ± 0.001.00 ± 0.001.00 ± 0.000.02 ± 0.01
ε\varepsilon-greedy1.00 ± 0.001.00 ± 0.001.00 ± 0.000.02 ± 0.01
RND1.00 ± 0.001.00 ± 0.001.00 ± 0.000.00 ± 0.00
Pseudocount0.18 ± 0.040.11 ± 0.030.05 ± 0.010.02 ± 0.01
Dynamics0.67 ± 0.330.34 ± 0.170.52 ± 0.140.01 ± 0.00
ECLAIR (ours)1.00 ± 0.001.00 ± 0.001.00 ± 0.000.15 ± 0.07

Table R2. Component ablations of ECLAIR (mean ± s.e. across 4 seeds). MultiRoom reports reward and Scene reports success rate. Scene-T1 values are from Table 3 of the paper. Bold indicates best performing method(s).

VariantMultiRoom-N4MultiRoom-N6Scene-T1Scene-T2
w/o online fine-tuning0.50 ± 0.10−0.22 ± 0.160.61 ± 0.180.65 ± 0.09
w/o KL clipping0.99 ± 0.000.93 ± 0.030.63 ± 0.210.68 ± 0.07
H=1H = 10.97 ± 0.020.98 ± 0.000.92 ± 0.030.59 ± 0.23
H=5H = 50.92 ± 0.050.91 ± 0.020.90 ± 0.040.64 ± 0.05
ECLAIR (ours, H=10H = 10)0.98 ± 0.010.90 ± 0.070.90 ± 0.040.88 ± 0.06

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.