acceptodds
Under review as a conference paper at ICLR 2027

Beyond Conservatism: Recoverability-Conditioned Exploration for Model-Based Imitation Learning

Abstract

Model-based imitation learning (MBIL) improves real-environment interaction efficiency by optimizing policies on imagined rollouts from a learned world model. However, the gap between model-induced and real-environment occupancies makes policy learning sensitive to model error. Conservative MBIL mitigates model exploitation during policy optimization, but when real-environment interactions are collected by the same conservative policy, uncertain regions around the expert distribution remain insufficiently sampled. Generic uncertainty-driven exploration, on the other hand, may allocate interaction to novel but task-irrelevant dynamics. We propose **RE**coverability-**CON**ditioned Exploration for Model-Based Imitation Learning (**RECON**). RECON separates conservative policy learning from active data collection by maintaining a main policy for task execution and an explorer for real-environment interaction. The explorer is optimized based on epistemic uncertainty conditioned on recoverability estimated from multi-step main-policy imagination, focusing data collection on unknown states from which the main policy can still return toward expert behavior. Experiments on locomotion, navigation and manipulation show consistent gains in interaction efficiency, imitation performance, and robustness, indicating that RECON directs real-environment interaction toward recovery regions around the expert distribution that are underexplored by prior methods, and thereby learns a world model better suited for imitation.

Primary area:
reinforcement learning
Keywords:
Imitation Learning, World Models, Model-Based Reinforcement Learning, Active Data Acquisition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9cfaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Model-based imitation learning algorithms offer sample-efficient solutions in the model-based setting by imagining trajectories based on a (world) model. Existing methods have a few limitations: conservative imitation learning will leave certain regions around the expert distribution to be insufficiently sampled; and purely uncertainty-driven exploration may explore task-irrelevant dynamics. The authors propose recoverability-conditioned exploration, where recovery is defined as the ability to return to expert behaviors and it is implemented for weighting an uncertainty component. The algorithm separates a conservative main policy which penalises the visitation of unknown states and an exploration policy which seeks out unknown states but can safely recover to known states. The results shows that across control, navigation, and manipulation benchmarks, this improves imitation, robustness and model accuracy.

Critical strengths

the results are promising and explainable:

  • the main policy is evaluated as a robustness test under action noise, delay, impulses, and dynamics shifts unseen during training
  • world model errors are much reduced compared to CMIL across many tasks
  • RECON shifts interaction toward near-support states and away from far-OOD novelty

The paper is well-presented and seems rigorous, though I could not check all the proofs.

Critical weaknesses
  • the related work discussion seems to be mostly limited to old papers before 2021
  • the related work discussion should also compare what theoretical guarantees are offered here and how they compare to other approaches
Questions for authors
  • Table 1: the notation in parentheses is not consistent
  • The analogues of model-based RL are important to discuss in related work.
    • For instance, there is a significant overlap of some of the ideas with E3 style algorithms in terms of the notion of unknown states and safe return, especially E4, which also forces agents to return to known states, and Neural E3, which is more similar in terms of representations. It seems one potential difference is that your algorithm uses specific temporal windows for the exploration policy. Any other relations seem interesting to mention.
    • Additionally, there is much theoretical development in that area and what seems to be missing is a comparison of guarantees offered. Can the authors discuss how favourable the theoretical guarantees are compared to other works?
  • more recent works should be included in the related works
  • the choice of dreamer V2. Why not V3 or V4?
  • eq. 13 gives opposite sign to the UθU_{\theta} terms, why?
  • th.1: significance of qq vs μ0\mu_0 notation of starting distribution
  • the authors mention the trajectories that can return to a known state in terms of a trust region, however trust region methods have a specific meaning in the parameter space. There is some correspondence between the trajectories implied by a policy and the policy. But I wonder why one would select one over the other. Perhaps the authors can elaborate on the trade-offs between the parameter space and state space exploration view.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 9cfa

Thank you for your constructive comments and insightful questions! We are glad to hear that you found the paper clearly presented and appreciated our robustness evaluation across unseen perturbations as well as the world-model improvements over CMIL. Following your suggestions, we have expanded the related-work and theoretical comparisons, clarified the notation and state-space perspective, and conducted additional DreamerV3 experiments in the revised manuscript. Below, we address each of your comments and questions in detail.

1. Relation to E³, E⁴, and Neural-E³, and comparison of theoretical guarantees.

We thank the reviewer for pointing out these connections. We have expanded Related Work and Appendices A.6–A.8 to compare how these methods identify unlearned dynamics, trigger exploration, and formulate guarantees.

E³: explore until the model is accurate enough for control. E³ partitions states into statistically known states and unknown states with insufficiently sampled successor dynamics. In known regions, the agent plans either to exploit the learned model or to reach an unknown state, collecting samples there until the model supports near-optimal control.

E⁴: reserve budget for robust escape. E⁴ extends E³ to constrained MDPs with explicit costs and budgets, separating exploitation in known regions, exploration toward unknown states, and escape back to known states. When further exploration risks exhausting the cost budget, the agent switches to an escape policy planned across plausible transition dynamics rather than relying on a familiar-looking endpoint. Under its CMDP feasibility and estimation assumptions, E⁴ guarantees polynomial-resource near-optimality and a high-probability bound on expected discounted exploration cost, Cπ(s)=E[∑t≥0γtc(st,at)∣s0=s]C^\pi(s)=\mathbb{E}\left[\sum_{t\geq 0}\gamma^t c(s_t,a_t)\mid s_0=s\right].

DREEM and Neural-E³: replace tabular knownness with multi-step model disagreement. DREEM extends explicit exploration to continuous spaces by maintaining candidate dynamics models consistent with observed data and planning a policy that maximizes divergence among their multi-step predicted state distributions. Executing this policy eliminates inaccurate candidates until the remaining agreement supports near-optimal exploitation, though finite neural ensembles can still agree on inaccurate predictions in unexplored regions.

Implications for RECON's positioning. RECON shares an important conceptual thread with these four works: all of them separate information acquisition from exploitation, direct exploration toward uncertain dynamics, and—most notably in E⁴—regulate exploration by whether the trajectory can return to a reliable region. RECON addresses a distinct problem specific to model-based imitation learning (MBIL): the policy is trained via imagined rollouts inside an imperfect world model to imitate expert trajectories rather than to maximize an external task reward. Because demonstrations cover only a narrow support, rollouts that drift outside it suffer from compounding model error; conservative MBIL suppresses model exploitation during policy learning, yet using that same conservative policy for environment collection starves the world model of transitions around recoverable deviations. RECON's distinct contribution is to decouple conservative model usage from active data acquisition in MBIL, conditioning exploration on multi-step recoverability under the current imitation policy toward expert-compatible states (rather than certifying tabular knownness or enforcing an explicit CMDP cost budget). Correspondingly, while E³, E⁴, and idealized DREEM establish end-to-end near-optimality under RL/CMDP assumptions, our Theorems 1–3 characterize how recovery-rollout error depends on data coverage and how a single query reduces recovery-weighted predictive risk. We thank the reviewer for highlighting this connection; we have integrated these comparisons into Related Work and Appendices A.6–A.8 to make the conceptual links, return criteria, and guarantee scope explicit.

2. Additional recent related work.

We have reorganized Related Work into four parallel parts: reinforcement and imitation learning, world models and conservative policy learning, model-based imitation learning, and exploration and data acquisition. The revision adds AIME-NoB, CDRED-WM, IL-SOAR, ILDE, OPT-AIL, recent exploration theory, and Dreamer 3/4. We separate directly relevant imitation-learning precursors from task-agnostic model-based RL methods and compare their acquisition mechanisms. Appendix A.6 provides a detailed comparison of return criteria and assumptions.

AuthorsOpenReview ↗

Response to Reviewer 9cfa(Part2)

3. Why DreamerV2 rather than DreamerV3 or DreamerV4?

DreamerV2 matches the backbone used by CMIL, isolating the effect of data acquisition; RECON itself is backbone-agnostic. We also upgraded both CMIL and RECON to DreamerV3 and ran three seeds per method on Hopper Stand and Walker Run (Appendix C.7, Table 7). Because hyperparameters are directly inherited from the default DreamerV2-based CMIL configuration without re-tuning for DreamerV3, the absolute returns under V3 are not directly comparable to those under V2; nevertheless, under the exact same settings and hyperparameters, as shown below across environment interaction steps (×105\times 10^5), RECON consistently outperforms CMIL on both tasks, demonstrating the generality and transferability of the RECON paradigm across MBIL world-model backbones. DreamerV4 trains agents from offline data with a scalable world model, an offline reinforcement-learning setting. Our MBIL loop instead alternates real interaction with model updates, so V4 is not a like-for-like backbone substitution; Related Work discusses it as a complementary line on scalable world models.

EnvironmentMethod0.5×1050.5\times 10^51×1051\times 10^51.5×1051.5\times 10^5
Hopper StandCMIL29.8 ± 37.9432.6 ± 469.6867.6 ± 76.5
Hopper StandRECON133.0 ± 184.3709.0 ± 149.6880.5 ± 49.4
EnvironmentMethod1.5×1051.5\times 10^52.5×1052.5\times 10^53.5×1053.5\times 10^5
Walker RunCMIL39.4 ± 10.865.9 ± 0.8134.1 ± 39.8
Walker RunRECON70.6 ± 73.4146.5 ± 102.7259.4 ± 147.3

4. Why does the uncertainty term in Equation 13 have the opposite sign?

This is intentional because the two policies serve different roles. The main policy subtracts uncertainty to discourage exploitation of unreliable imagined transitions. The exploration policy receives a positive uncertainty reward weighted by predicted recoverability, directing real interaction toward under-specified dynamics that remain relevant to possible recovery by the main policy. Both policies retain the imitation signal. Section 4 explains this distinction between model use and model acquisition directly after the objectives.

5. Meaning of qq and μ0\mu_0.

μ0\mu_0 is the environment's normal reset distribution, while qq is a start-state distribution used to analyze recovery. Dynamics and policy are held fixed; changing the start distribution lets us examine whether normal data cover states visited during recovery under the same policy. The learner is not given the ability to reset the environment from arbitrary qq. Section 3 defines both distributions before use, and Appendix B.5 clarifies that persistent action noise or changes in dynamics cannot be analyzed solely by changing the start distribution.

6. Model confidence regions and policy-space exploration.

Thank you for identifying the ambiguity in the term trust region [1]. We have replaced it with model confidence region to denote the state–action subset where ensemble uncertainty is low and world-model predictions are reliable, and removed phrasing that could suggest a parameter-space constraint on policy updates.

As you note, a policy together with μ0\mu_0 and the dynamics induces a trajectory distribution, but parameter distance does not directly determine which state–action transitions are visited. The mapping from parameters to trajectories is many-to-one [2][3], and near contact or bifurcation boundaries small action perturbations can lead to sharply different states; conversely, two policies can match on nominal expert support while diverging once perturbed to a recovery start distribution qq.

We adopt the state–action view because RECON decides where to spend a limited real-interaction budget to improve a local dynamics model. Whether a transition is worth measuring depends on its local epistemic uncertainty and on whether the trajectory can return toward expert-compatible states under the main policy—properties defined along visited state–action trajectories rather than by the size of a parameter step.

The two views have complementary trade-offs: parameter-space exploration provides smooth behavioral control and temporally coherent rollouts from a single parameter vector without multi-step imagination [4][5], whereas state-space evaluation directly filters transitions by local uncertainty and predicted recoverability at the cost of finite-horizon imagined rollouts. Appendix A.8 formalizes this comparison via an occupancy-weighted Fisher metric and an out-of-support counterexample.

AuthorsOpenReview ↗

Response to Reviewer 9cfa(Part3)

7. Parenthetical notation in Table 1.

The values in parentheses were absolute differences: return differences for Hopper and Walker Run, and percentage-point differences for success-rate tasks. We have removed this potentially ambiguous notation and explain the units in the caption. The full-baseline table in Appendix C.8 reports means and dispersion directly and no longer uses parenthetical gains. The caption specifies the units, with bold marking the highest mean, including ties.

We sincerely thank you again for prompting us to reflect on these deeper theoretical connections and for your helpful feedback on the manuscript. Hope these can address your concerns!

References

[1] Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust Region Policy Optimization. ICML, PMLR 37, 1889–1897.

[2] Parker-Holder, J., Pacchiano, A., Choromanski, K., & Roberts, S. (2020). Effective Diversity in Population Based Reinforcement Learning. NeurIPS, 33, 18050–18062.

[3] Müller, J., Çaycı, S., & Montúfar, G. (2025). Fisher–Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients. SIAM Journal on Optimization, 35(2), 1060–1088.

[4] Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., & Andrychowicz, M. (2018). Parameter Space Noise for Exploration. ICLR 2018 (arXiv:1706.01905).

[5] Chiappa, A. S., Marin Vargas, A., Huang, A., & Mathis, A. (2023). Latent Exploration for Reinforcement Learning. NeurIPS, 36, 56508–56530.

Reviewer_BgveOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose RECON, an extension of Conservative Model-Based Imitation Learning that introduces an additional exploration policy. This exploration policy aims to visit uncertain yet recoverable states that the conservative main policy would otherwise avoid. Recoverability is estimated through multi-step imagination of the main policy and the learned world model. By alternating between the exploration and main policies, RECON seeks to explore states with high model uncertainty while ensuring that the main policy can recover from the resulting deviations. The authors evaluate RECON on eight tasks including locomotion, navigation, and manipulation, demonstrating improved imitation learning performance and robustness compared to existing baselines.

Critical strengths
  • The problem formulation is well motivated. The distinction between the conservative policy for deployment and exploration policy for data acquisition is meaningful. Improving the world model near recoverable area is a sensible objective for imitation learning.
  • The improvement over CMIL is significant. While keeping most of the algorithm design of CMIL, the involvement of an exploration policy greatly improve the policy robustness as shown in Tab.1.
Critical weaknesses
  • The improvements over existing methods are not clearly demonstrated. As shown in Fig. 3, RECON does not significantly outperform existing baselines on most tasks. Furthermore, only CMIL and RECON are included in the quantitative comparisons of per-task performance and robustness, making it difficult to assess RECON's advantages over other baselines.
Questions for authors
  1. How does RECON compare with existing exploration methods in MBIL?
  2. Can the authors clarify the evaluation protocol underlying Tab.1? In particular, why do both policies perform the worst in clean environment of Hammer?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Bgve

We sincerely thank you for your thoughtful review and valuable suggestions! We are glad to hear that you found our motivation for decoupling conservative deployment from active data acquisition well-grounded. Your comments also prompted us to broaden our empirical comparisons across all baselines, add adapted MBIL exploration controls, and audit the counterintuitive Hammer evaluation behavior in depth. Below, we address your questions and concerns point by point.

1. Per-task performance and robustness across baselines.

Appendix C.8 (Table 8) reports V-MAIL, IQ-MPC, DA-DAC, CMIL, and RECON under the nominal and perturbed deployment protocol in Appendix B.5. All methods use final checkpoints from three seeds. RECON has the highest reported mean in every condition on Hopper Stand, Walker Run, and PointMaze, compared with all baselines. On Walker Run, for example, RECON reaches 593.9 ± 90.3 under clean conditions and 514.6 ± 91.9 under impulse/scale perturbations; the corresponding V-MAIL results are 204.9 ± 38.6 and 122.3 ± 21.0. On Hopper Stand with delay, RECON obtains 596.2, compared with 408.5 for CMIL and 190.3 for V-MAIL. On PointMaze with delay, RECON reaches 100.0%, compared with 30.3% for CMIL.

From an inductive-bias perspective, RECON is not expected to yield a uniform gain in nominal imitation performance or sample efficiency on every task, because it intentionally allocates part of the interaction budget to exploring recoverable dynamics in the neighborhood of expert trajectories. On tasks with relatively simple dynamics (such as Handle Press or Faucet Close), standard conservative or single-policy methods concentrate their entire budget on fitting the nominal expert trajectory within the conservative region, and can therefore match or occasionally exceed RECON in nominal performance. Under deployment perturbations, however, purely conservative methods degrade sharply because the world model lacks coverage of nearby recovery dynamics, whereas RECON achieves a more favorable trade-off between nominal imitation and robustness (e.g., on Faucet Close under delay, CMIL drops to 5.0% while RECON reaches 85.0%, when RECON matches or even partially exceeds CMIL in interaction efficiency on this task). In systems with more complex, underactuated dynamics (such as Walker Run and Hopper Stand), where even small rollout errors require continual recovery, RECON leads across both nominal imitation performance and perturbed robustness. Both our main experiments and the full-baseline robustness evaluation in Appendix C.8 (Table 8) support this interpretation.

2. Comparison with exploration methods in MBIL and imitation learning.

Thank you for raising this question. We would like to clarify why active exploration in MBIL differs fundamentally from exploration in model-free IL and model-based RL. Exploration has been widely studied in model-free IL (e.g., optimistic critic estimation and double exploration in IL-SOAR, ILDE, and OPT-AIL) and in model-based RL (e.g., E³, E⁴, DREEM/Neural-E³, MAX, and Plan2Explore). In model-based IL, however, the policy interacts with an imperfect learned world model through multi-step imagined rollouts, and its goal is solely to imitate expert trajectories rather than to discover high-reward behaviors across the state space. Encouraging the imitation policy to seek uncertain states inside imagined rollouts invites model exploitation and compounding rollout error, degrading imitation performance—a core challenge documented across world-model imitation and conservative model-based learning (e.g., V-MAIL, MOPO, MOReL, CMIL, and AIME-NoB). Consequently, modern online MBIL predominantly adopts a conservative paradigm such as CMIL.

Our key observation is that this conservative paradigm overlooks the decoupling between the usage question (which model predictions are safe for policy optimization?) and the acquisition question (which real transitions should be collected to improve the world model?). When a single conservative policy serves both roles, penalizing uncertainty to prevent model exploitation simultaneously shrinks real data collection around uncertain recovery states. RECON decouples these roles: the main policy remains conservative, while a separate explorer acquires task-relevant real transitions. Specifically, we adapt Plan2Explore's imagined-disagreement explorer to MBIL by retaining discriminator guidance for task relevance, weighting uncertainty by multi-step recoverability under the current main policy (re=Dψ+βUθGr_e = D_\psi + \beta U_\theta G), and alternating imagined updates with real collection on a shared world model. Our focus is this usage–acquisition decoupling paradigm rather than a specific exploration bonus; UθU_\theta can structurally be replaced by other bonuses b(s,a)b(s,a) (such as RND) via re=Dψ+βb(s,a)Gr_e = D_\psi + \beta b(s,a)G in future work.

(continued in next response)

AuthorsOpenReview ↗

Response to Reviewer Bgve(Part2)

For the reasons above, existing MBIL methods have generally overlooked exploration within the data-acquisition problem, and there is no standard off-the-shelf exploration baseline tailored to online MBIL. To evaluate transferring an established MBRL exploration paradigm directly into MBIL, we added two Plan2Explore-style controls on Hopper Stand and Walker Run over 3 seeds with the same environment-interaction budget as the main experiments (Appendix C.6): (i) Control 1, which removes discriminator guidance and recovery gating to collect data online with pure ensemble disagreement (re=βUθr_e = \beta U_\theta); and (ii) Control 2, which first pretrains the world model using a disagreement explorer and then performs downstream imitation. As shown below, both fail to improve policy performance, confirming that effective exploration in MBIL requires task guidance and an interleaved online data-acquisition paradigm, while the ablations in Figure 5 further establish the necessity of recoverability conditioning.

EnvironmentFinal returnMaximum return
Control 1: pure-disagreement online acquisition
Hopper Stand3.49 ± 3.2417.36 ± 12.43
Walker Run16.79 ± 4.7728.19 ± 1.41
Control 2: exploratory pretraining, then imitation
Hopper Stand37.41 ± 35.26138.61 ± 12.27
Walker Run53.66 ± 19.04118.84 ± 32.63

We have incorporated a systematic discussion of these related exploration lines—covering online MBIL acquisition (MobILE, AIME-NoB, CDRED-WM), optimistic IL (IL-SOAR, ILDE, OPT-AIL), and model-based RL exploration (E³, E⁴, DREEM/Neural-E³)—into the revised Related Work.

3. Evaluation protocol underlying Table 1.

Appendix B.5 now specifies the protocol. Final checkpoints from three training seeds are held fixed across perturbations; for RECON, the explorer is disabled and only the deterministic main policy acts. Noise uses Gaussian action perturbations with standard deviations 0.050.05 and 0.100.10, and delay uses one- and two-step action delays. The Impulse/scale column aggregates separately evaluated impulse, action-dropout, action-scaling, and mass/friction configurations. Actions are clipped to their valid range. We first average 50 episodes within each configuration, then configurations within a condition, and finally training seeds. Hopper Stand and Walker Run report episode return; the remaining tasks report success rate.

4. Why Hammer can perform worse under clean conditions.

Thank you for highlighting this counterintuitive result. We traced it to the Meta-World Hammer success criterion. The current environment marks an episode as successful when NailSlideJoint > 0.09, without requiring the agent to actually pick up or use the hammer. Our manual inspection found that some success-labeled trajectories slide the nail in directly using the robot arm or gripper. This is benchmark-level reward hacking: the agent satisfies the proxy success signal without completing the intended task.

Because the evaluator marks an episode successful if the success flag is observed at any step, perturbations that induce variations in action magnitude, shifts in contact timing, or offsets in the gripper's approach trajectory and contact pose can create more opportunities to accidentally trigger this shortcut. This behavior does not come from optimizing the environment reward, since all baselines are trained from expert actions. Appendix C.5 documents the evidence and its limitations. We plan to validate a revised success criterion and then submit a PR to Meta-World to fix this evaluation issue and provide the RL community with a more reliable Hammer benchmark.

Thank you again for your time and constructive feedback, which have substantially strengthened our experimental evaluation and analysis. We hope these clarifications and additional results address your concerns, and we would be grateful if you would consider raising your score!

Thank you for the detailed response and additional experiments. The expanded robustness comparisons and Plan2Explore-style baselines help address my main concerns. I also appreciate the clarification of the evaluation protocol and the investigation into the Hammer benchmark issue.

While the exploration comparisons remain limited to two tasks, the additional evidence strengthens the paper. I am currently leaning toward raising my score, but will make a final decision during the reviewer AC discussion phase.

Reviewer_sehJOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an exploration mechanism based on recoverability to make better use of limited real-world interaction in model-based imitation learning. Next to a conservative main policy, a second policy tries to explore states with high model uncertainty that are still being recoverable from by the main policy. A defined portion of the environment interaction is used by the exploration policy solely for model improvement. Recoverability of states is determined by rolling out the main policy using the learned model. The method improves on prior work by focusing exploration and model improvement on states around the expert trajectory instead of potentially irrelevant state space regions with high model uncertainty. The authors evaluate their method mostly through a direct comparison to CMIL, which does not make use of the recoverability concept. Other baselines are evaluated to compare imitation performance. The results show minor improvements in plain imitation performance but a clear improvement in policy robustness. No AI was used in writing this review.

Critical strengths

The paper proposes a well-motivated exploration mechanism to increase policy robustness and model improvement efficiency in MBIL. The conducted experiments include a variety of different tasks and convey the benefits of the proposed method while being transparent about moderate gains in some tasks and metrics, allowing for an objective assessment.

Critical weaknesses

The paper lacks clarity in some sections, especially in section 3, where notation is not properly introduced and in section 5, where the posed questions are not clearly distinguished. Improvement over prior work is incremental and the impact of the proposed method likely moderate.

Questions for authors

As a central claim is improved efficiency in terms of needed environment interaction, it would be interesting to see an experiment that measures how much data is needed for different baselines to reach the same policy performance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer sehJ

Thank you very much for your detailed review and constructive suggestions! We are pleased to hear that you found the motivation well-founded and our experimental evaluation broad. We especially appreciate your suggestion to measure the environment interactions needed to reach matched performance thresholds, which provides a much more direct test of sample efficiency. Below, we follow your suggestions to revise the manuscript and address each of your concerns.

1. Real interactions required to reach matched performance.

The horizontal axis of Figure 3 counts environment interaction, not imagined transitions. In the Dreamer training loop, every five real transitions trigger two model/actor–critic update cycles; imagined rollouts do not increase the real-interaction count. Comparing horizontal positions at the same vertical performance level in Figure 3 therefore directly compares the real interactions required across methods. To present this more clearly, Appendix C.4 gives the matched-threshold analysis, and the new threshold tables report actual env.step transitions only. For each task, let Jmax⁡J_{\max} be the highest final three-seed mean among all methods in the Figure 3 training logs. We report the number of executed env.step transitions at which each method's three-seed mean curve first reaches 25%, 50%, 75%, and 100% of Jmax⁡J_{\max}. Notable comparisons include:

Task and common targetRECONComparisonFewer real transitions
Hopper Stand, 50%45KV-MAIL: 55K18.2%
Hopper Stand, 75%57.5KCMIL: 75K23.3%
PointMaze, 50%75KCMIL: 100K25.0%
Walker Run, 25%95KCMIL: 135K29.6%

The complete tables are included in Table 5.

Figure 9 explains why the acquired data can be more useful. Appendix C.2 measures the uncertainty reduction per newly acquired real transition, RECON has the highest acquisition gain in this diagnostic. An uncertainty-only collector is not even as effective as V-MAIL's imitation-reward collector. Seeking uncertain transitions alone therefore need not improve the predictions that the imitation policy actually uses. The discriminator keeps acquisition related to expert behavior, and the recovery weight prioritizes under-specified transitions from which the main policy is predicted to recover. These real transitions train the shared world model through its prediction objective and require no expert recovery-action labels.

Notably, from an inductive-bias perspective, RECON is not expected to yield a uniform gain in nominal imitation performance or sample efficiency on every task, because it intentionally allocates part of the interaction budget to exploring recoverable dynamics in the neighborhood of expert trajectories. On tasks with relatively simple dynamics (such as Handle Press or Faucet Close), standard conservative or single-policy methods concentrate their entire budget on fitting the nominal expert trajectory within the conservative region, and can therefore match or occasionally exceed RECON in nominal performance. Under deployment perturbations, however, purely conservative methods degrade sharply because the world model lacks coverage of nearby recovery dynamics, whereas RECON achieves a more favorable trade-off between nominal imitation and robustness (e.g., on Faucet Close under delay, CMIL drops to 5.0% while RECON reaches 85.0%,when RECON matches or even partially exceeds CMIL in interaction efficiency on this task). In systems with more complex, underactuated dynamics (such as Walker Run and Hopper Stand), where even small rollout errors require continual recovery, RECON leads across both nominal imitation performance and perturbed robustness. Both our main experiments and the full-baseline robustness evaluation in Appendix C.8 (Table 8) support this interpretation.

2. Clarity of Section 3 and organization of Section 5.

Section 3 now introduces the normal reset distribution μ0\mu_0, recovery-start distribution qq, occupancy distributions, reference distribution ν\nu, gates gkg_k, and weighted risk before they are used. It also distinguishes the per-transition gates in the analysis from the shared terminal recovery score used in the Section 4 implementation. Appendix A expands the coverage condition, residual error, and information transfer, and states the relevant approximations and guarantee scope.

(continued in next response)

AuthorsOpenReview ↗

Response to Reviewer sehJ(Part2)

Section 5 now connects each question directly to its experiment and evidence:

ParagraphQuestionEvidence
5(a)Does acquisition improve imitation under a fixed interaction budget?Figure 3
5(b)How robust is the deployed policy to perturbations?Table 1; full baselines in Appendix C.8
5(c)Where does recoverability redirect real interaction?Figure 4
5(d)Which acquisition signals and design choices matter?Figures 5–6
5(e)Does targeted acquisition improve the relevant model?Figure 7 and Table 2

The matched-performance analysis, Plan2Explore-style controls, and DreamerV3 comparison appear in Appendices C.4, C.6, and C.7, respectively.

3. Scope and magnitude of the contribution.

We thank the reviewer for these relevant comments. We would like to further clarify our contribution. RECON deliberately retains CMIL’s conservative main-policy objective and backbone, so that the comparison primarily isolates the effect of the real-environment data-acquisition mechanism. Our central observation is that conservative MBIL typically lets a single policy implicitly address two distinct questions: which model predictions are safe to use for policy optimization (Usage question), and which unknown dynamics should be measured next under a limited interaction budget (Acquisition question). This coupling causes mechanisms intended to suppress model error to also narrow the scope of data acquisition, making recovery-relevant uncertainty difficult to resolve actively. This issue, however, remains largely underexamined in the literature.

RECON separates these decisions and uses recoverability under the current imitation policy to connect task relevance with information value. This yields a general design principle: acquisition should target uncertainty that can change the policy’s ability to recover, with novelty or global model error serving as auxiliary signals. The principle is independent of a particular uncertainty estimator and can inform task-aware exploration in model-based RL, data acquisition in low-interaction imitation learning and robust deployment. We therefore believe that the core insight of this paper can broaden the research perspective for future work in these related areas.

We sincerely appreciate your constructive feedback on both the empirical evaluation and the clarity of Sections 3 and 5, which has made the revised paper much easier to follow. Thank you again for your time and support!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Response — Thank You to the Reviewers and Area Chair

We would like to sincerely thank all three reviewers (Reviewer sehJ, Reviewer Bgve, and Reviewer 9cfa) and the Area Chair for their time, thoughtful feedback, and careful consideration of our work throughout the review and discussion process.

We are especially grateful for the constructive and open-minded discussions. The reviewers' questions encouraged us to examine RECON more rigorously, sharpen the presentation of our core usage–acquisition question decoupling insight, and broaden the theoretical and empirical evidence supporting our claims. In response, we substantially revised the manuscript (Sections 2–5) and added Appendices A.6–A.8, B.5, and C.4–C.8 (Tables 5–8), including:

  • Stronger theoretical positioning and conceptual clarity: Expanded comparisons with explicit model-based exploration (E3E^3, E4E^4, and DREEM/Neural-E3E^3) and their guarantee scopes (Appendices A.6–A.7), a formal analysis of parameter-space versus state-space exploration via an occupancy-weighted Fisher metric and an out-of-support counterexample (Appendix A.8), and clearer definitions of the start distributions (μ0,q\mu_0, q), model confidence regions, and exploration–deployment objectives in Sections 3–4.
  • More comprehensive empirical validation: Full-baseline robustness evaluations across all eight environments under nominal and perturbed conditions (Appendix C.8, Table 8), matched-performance sample-efficiency analyses in terms of actual env.step transitions combined with per-transition uncertainty-reduction diagnostics (Appendices C.2 and C.4, Table 5), two Plan2Explore-style controls across three domains testing unconstrained online disagreement and exploratory pretraining with offline imitation (Appendix C.6, Table 6), and backbone-transfer experiments upgrading CMIL and RECON to DreamerV3 (Appendix C.7, Table 7).
  • Improved reproducibility, benchmark diagnostics, and literature positioning: A detailed specification of the multi-condition perturbation evaluation protocol (Appendix B.5), a trajectory-level audit identifying benchmark-level reward hacking in the Meta-World Hammer success criterion (Appendix C.5), and a reorganized Related Work (Section 2) covering recent advances in world models, online MBIL acquisition, and optimistic imitation learning.

Beyond addressing individual questions, we believe these revisions and analyses have helped us present a much more rigorous, transparent, and complete account of RECON's contributions.

We particularly appreciate the reviewers' positive recognition of our clarifications and supplementary results, and their willingness to engage with our responses and additional evidence during the discussion period.Their careful evaluations and constructive suggestions have been invaluable in strengthening this work.

Finally, we thank the Area Chair for overseeing the evaluation process and considering the reviews, rebuttals, and discussions as a whole.

Regardless of the final decision, we are genuinely grateful for the opportunity to improve RECON through this exchange. Thank you again for your time, expertise, and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.