- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Model-based imitation learning algorithms offer sample-efficient solutions in the model-based setting by imagining trajectories based on a (world) model. Existing methods have a few limitations: conservative imitation learning will leave certain regions around the expert distribution to be insufficiently sampled; and purely uncertainty-driven exploration may explore task-irrelevant dynamics. The authors propose recoverability-conditioned exploration, where recovery is defined as the ability to return to expert behaviors and it is implemented for weighting an uncertainty component. The algorithm separates a conservative main policy which penalises the visitation of unknown states and an exploration policy which seeks out unknown states but can safely recover to known states. The results shows that across control, navigation, and manipulation benchmarks, this improves imitation, robustness and model accuracy.
Critical strengths
the results are promising and explainable:
- the main policy is evaluated as a robustness test under action noise, delay, impulses, and dynamics shifts unseen during training
- world model errors are much reduced compared to CMIL across many tasks
- RECON shifts interaction toward near-support states and away from far-OOD novelty
The paper is well-presented and seems rigorous, though I could not check all the proofs.
Critical weaknesses
- the related work discussion seems to be mostly limited to old papers before 2021
- the related work discussion should also compare what theoretical guarantees are offered here and how they compare to other approaches
Questions for authors
- Table 1: the notation in parentheses is not consistent
- The analogues of model-based RL are important to discuss in related work.
- For instance, there is a significant overlap of some of the ideas with E3 style algorithms in terms of the notion of unknown states and safe return, especially E4, which also forces agents to return to known states, and Neural E3, which is more similar in terms of representations. It seems one potential difference is that your algorithm uses specific temporal windows for the exploration policy. Any other relations seem interesting to mention.
- Additionally, there is much theoretical development in that area and what seems to be missing is a comparison of guarantees offered. Can the authors discuss how favourable the theoretical guarantees are compared to other works?
- more recent works should be included in the related works
- the choice of dreamer V2. Why not V3 or V4?
- eq. 13 gives opposite sign to the terms, why?
- th.1: significance of vs notation of starting distribution
- the authors mention the trajectories that can return to a known state in terms of a trust region, however trust region methods have a specific meaning in the parameter space. There is some correspondence between the trajectories implied by a policy and the policy. But I wonder why one would select one over the other. Perhaps the authors can elaborate on the trade-offs between the parameter space and state space exploration view.
5 comments
Response to Reviewer 9cfa(Part2)
3. Why DreamerV2 rather than DreamerV3 or DreamerV4?
DreamerV2 matches the backbone used by CMIL, isolating the effect of data acquisition; RECON itself is backbone-agnostic. We also upgraded both CMIL and RECON to DreamerV3 and ran three seeds per method on Hopper Stand and Walker Run (Appendix C.7, Table 7). Because hyperparameters are directly inherited from the default DreamerV2-based CMIL configuration without re-tuning for DreamerV3, the absolute returns under V3 are not directly comparable to those under V2; nevertheless, under the exact same settings and hyperparameters, as shown below across environment interaction steps (), RECON consistently outperforms CMIL on both tasks, demonstrating the generality and transferability of the RECON paradigm across MBIL world-model backbones. DreamerV4 trains agents from offline data with a scalable world model, an offline reinforcement-learning setting. Our MBIL loop instead alternates real interaction with model updates, so V4 is not a like-for-like backbone substitution; Related Work discusses it as a complementary line on scalable world models.
| Environment | Method | |||
|---|---|---|---|---|
| Hopper Stand | CMIL | 29.8 ± 37.9 | 432.6 ± 469.6 | 867.6 ± 76.5 |
| Hopper Stand | RECON | 133.0 ± 184.3 | 709.0 ± 149.6 | 880.5 ± 49.4 |
| Environment | Method | |||
| Walker Run | CMIL | 39.4 ± 10.8 | 65.9 ± 0.8 | 134.1 ± 39.8 |
| Walker Run | RECON | 70.6 ± 73.4 | 146.5 ± 102.7 | 259.4 ± 147.3 |
4. Why does the uncertainty term in Equation 13 have the opposite sign?
This is intentional because the two policies serve different roles. The main policy subtracts uncertainty to discourage exploitation of unreliable imagined transitions. The exploration policy receives a positive uncertainty reward weighted by predicted recoverability, directing real interaction toward under-specified dynamics that remain relevant to possible recovery by the main policy. Both policies retain the imitation signal. Section 4 explains this distinction between model use and model acquisition directly after the objectives.
5. Meaning of and .
is the environment's normal reset distribution, while is a start-state distribution used to analyze recovery. Dynamics and policy are held fixed; changing the start distribution lets us examine whether normal data cover states visited during recovery under the same policy. The learner is not given the ability to reset the environment from arbitrary . Section 3 defines both distributions before use, and Appendix B.5 clarifies that persistent action noise or changes in dynamics cannot be analyzed solely by changing the start distribution.
6. Model confidence regions and policy-space exploration.
Thank you for identifying the ambiguity in the term trust region [1]. We have replaced it with model confidence region to denote the state–action subset where ensemble uncertainty is low and world-model predictions are reliable, and removed phrasing that could suggest a parameter-space constraint on policy updates.
As you note, a policy together with and the dynamics induces a trajectory distribution, but parameter distance does not directly determine which state–action transitions are visited. The mapping from parameters to trajectories is many-to-one [2][3], and near contact or bifurcation boundaries small action perturbations can lead to sharply different states; conversely, two policies can match on nominal expert support while diverging once perturbed to a recovery start distribution .
We adopt the state–action view because RECON decides where to spend a limited real-interaction budget to improve a local dynamics model. Whether a transition is worth measuring depends on its local epistemic uncertainty and on whether the trajectory can return toward expert-compatible states under the main policy—properties defined along visited state–action trajectories rather than by the size of a parameter step.
The two views have complementary trade-offs: parameter-space exploration provides smooth behavioral control and temporally coherent rollouts from a single parameter vector without multi-step imagination [4][5], whereas state-space evaluation directly filters transitions by local uncertainty and predicted recoverability at the cost of finite-horizon imagined rollouts. Appendix A.8 formalizes this comparison via an occupancy-weighted Fisher metric and an out-of-support counterexample.
Response to Reviewer 9cfa(Part3)
7. Parenthetical notation in Table 1.
The values in parentheses were absolute differences: return differences for Hopper and Walker Run, and percentage-point differences for success-rate tasks. We have removed this potentially ambiguous notation and explain the units in the caption. The full-baseline table in Appendix C.8 reports means and dispersion directly and no longer uses parenthetical gains. The caption specifies the units, with bold marking the highest mean, including ties.
We sincerely thank you again for prompting us to reflect on these deeper theoretical connections and for your helpful feedback on the manuscript. Hope these can address your concerns!
References
[1] Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust Region Policy Optimization. ICML, PMLR 37, 1889–1897.
[2] Parker-Holder, J., Pacchiano, A., Choromanski, K., & Roberts, S. (2020). Effective Diversity in Population Based Reinforcement Learning. NeurIPS, 33, 18050–18062.
[3] Müller, J., Çaycı, S., & Montúfar, G. (2025). Fisher–Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients. SIAM Journal on Optimization, 35(2), 1060–1088.
[4] Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2018). Parameter Space Noise for Exploration. ICLR 2018 (arXiv:1706.01905).
[5] Chiappa, A. S., Marin Vargas, A., Huang, A., & Mathis, A. (2023). Latent Exploration for Reinforcement Learning. NeurIPS, 36, 56508–56530.
Response to Reviewer 9cfa
Thank you for your constructive comments and insightful questions! We are glad to hear that you found the paper clearly presented and appreciated our robustness evaluation across unseen perturbations as well as the world-model improvements over CMIL. Following your suggestions, we have expanded the related-work and theoretical comparisons, clarified the notation and state-space perspective, and conducted additional DreamerV3 experiments in the revised manuscript. Below, we address each of your comments and questions in detail.
1. Relation to E³, E⁴, and Neural-E³, and comparison of theoretical guarantees.
We thank the reviewer for pointing out these connections. We have expanded Related Work and Appendices A.6–A.8 to compare how these methods identify unlearned dynamics, trigger exploration, and formulate guarantees.
E³: explore until the model is accurate enough for control. E³ partitions states into statistically known states and unknown states with insufficiently sampled successor dynamics. In known regions, the agent plans either to exploit the learned model or to reach an unknown state, collecting samples there until the model supports near-optimal control.
E⁴: reserve budget for robust escape. E⁴ extends E³ to constrained MDPs with explicit costs and budgets, separating exploitation in known regions, exploration toward unknown states, and escape back to known states. When further exploration risks exhausting the cost budget, the agent switches to an escape policy planned across plausible transition dynamics rather than relying on a familiar-looking endpoint. Under its CMDP feasibility and estimation assumptions, E⁴ guarantees polynomial-resource near-optimality and a high-probability bound on expected discounted exploration cost, Cπ(s)=E[∑t≥0γtc(st,at)∣s0=s].
DREEM and Neural-E³: replace tabular knownness with multi-step model disagreement. DREEM extends explicit exploration to continuous spaces by maintaining candidate dynamics models consistent with observed data and planning a policy that maximizes divergence among their multi-step predicted state distributions. Executing this policy eliminates inaccurate candidates until the remaining agreement supports near-optimal exploitation, though finite neural ensembles can still agree on inaccurate predictions in unexplored regions.
Implications for RECON's positioning. RECON shares an important conceptual thread with these four works: all of them separate information acquisition from exploitation, direct exploration toward uncertain dynamics, and—most notably in E⁴—regulate exploration by whether the trajectory can return to a reliable region. RECON addresses a distinct problem specific to model-based imitation learning (MBIL): the policy is trained via imagined rollouts inside an imperfect world model to imitate expert trajectories rather than to maximize an external task reward. Because demonstrations cover only a narrow support, rollouts that drift outside it suffer from compounding model error; conservative MBIL suppresses model exploitation during policy learning, yet using that same conservative policy for environment collection starves the world model of transitions around recoverable deviations. RECON's distinct contribution is to decouple conservative model usage from active data acquisition in MBIL, conditioning exploration on multi-step recoverability under the current imitation policy toward expert-compatible states (rather than certifying tabular knownness or enforcing an explicit CMDP cost budget). Correspondingly, while E³, E⁴, and idealized DREEM establish end-to-end near-optimality under RL/CMDP assumptions, our Theorems 1–3 characterize how recovery-rollout error depends on data coverage and how a single query reduces recovery-weighted predictive risk. We thank the reviewer for highlighting this connection; we have integrated these comparisons into Related Work and Appendices A.6–A.8 to make the conceptual links, return criteria, and guarantee scope explicit.
2. Additional recent related work.
We have reorganized Related Work into four parallel parts: reinforcement and imitation learning, world models and conservative policy learning, model-based imitation learning, and exploration and data acquisition. The revision adds AIME-NoB, CDRED-WM, IL-SOAR, ILDE, OPT-AIL, recent exploration theory, and Dreamer 3/4. We separate directly relevant imitation-learning precursors from task-agnostic model-based RL methods and compare their acquisition mechanisms. Appendix A.6 provides a detailed comparison of return criteria and assumptions.