acceptodds
Withdrawn from ICLR 2027

Learn from Failure, Act with Memory: Reflective and Prospective Learning for Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploration–exploitation coordination is a core challenge in reinforcement learning. Existing methods rely on static weighting or annealing variants, limiting state-wise exploration–exploitation balance. This causes premature convergence or ineffective exploration, impairing learning efficiency and performance. Essentially, the coordination problem is not about static allocation but an experience-based decision problem requiring agents to remember, reflect on, and leverage past successes and failures. Inspired by human retrospective memory and prospective anticipation, we propose an integrated adaptive exploration–exploitation framework, . On the exploitation side, the Retrospective Module distills shared patterns from successes and identifies weaknesses from failures for targeted improvement. On the exploration side, the Prospective Module combines coarse global search with fine-grained state discrimination via sparsity-constrained latent representations. We theoretically justify proposed exploration mechanism. Furthermore, we introduce a policy-level ensemble to adaptively coordinate the prospective and retrospective modules, enabling a exploration–exploitation balance based on demands. We prove the coordinated strategy outperforms non-integrated strategies. can be plugged into existing algorithms. We evaluate across 56 environments against 23 baselines in total, where our method achieves better average performance than baselines. Notably, it achieves up to a 15× improvement on Montezuma’s Revenge. improves time efficiency over baselines by 41.8% on average.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
learning on graphs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AaZzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new plug-an-play exploration strategy for RL algorithms named MemoRL. The proposed approach stores and learns valuable trajectories (high relative returns + relatively short). It uses BC as intrinsic reward to reinforce the RL policy over these trajectories. Theorem 1 claims the intrinsic reward keeps the optimal action unchanged. Theorem 2 claims the gated policy is at least as good as the base policy. The approach is validate on a diverse set of algorithms and domains.

Critical strengths
  • The proposed method is extensively and thoroughly evaluated on various RL algorithms and domains. Results are strong on hard exploration domains.

  • Learning from failures (not only from successful trajectories) is novel to the best of my knowledge and the ablation study shows that doing so is beneficial.

Critical weaknesses
  • Missing baselines and related work. 1. Go-Explore, 2. Agent57 (cited in the appendix as an intrinsic-motivation example), 3. Ensemble-UCB gating, 4. #Exploration and E3B, 5. Episodic control. These should all be compared against or at least discussed.

  • Tables 3 and 4 report no source, budget, seeds or variance. Some entries, such as the missing CR-SAC HalfCheetah result, were taken from the original papers? It is unclear whether their settings match MemoRL’s.

  • The abstract claims "it achieves up to a 15× improvement on Montezuma’s Revenge." but that is somewhat misleading because it is over basic PPO. Including RND already brings this down to x3.7.

  • Seeds and confidence intervals are missing for MuJoCo and Figs. 10 and 11.

  • PPO is reported as “440.3 ± 0” over 5 seeds.

  • Table 1 averages 5 games, while the text says 12.

  • No experiment shows that ∥fθ(s)∥1\|f_\theta(s)\|_1 tracks importance or novelty. The difference from Yan et al. (2023; 2024) is not stated.

  • Theorem 1 assumes the bonus is below the smallest action gap, which is near zero in Atari. Theorem 2 assumes πθ\pi_\theta is greedy with respect to QπQ^\pi, which behavior cloning does not ensure.

Questions for authors
  1. How is “failure” defined in tasks without terminal death states (e.g., Swimmer, Reacher)? How do the memory networks handle continuous actions?

  2. How does your adaptive coordination differ from Agent57’s meta-controller? How does the Prospective Module differ from Yan et al. (2023; 2024)?

Reviewer_RQpHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MemoRL, a reinforcement learning framework that combines retrospective memory, prospective exploration, and adaptive policy selection. The retrospective module learns a memory-based policy from high-return trajectories and trains a reflection network to identify potentially unsuccessful actions. The prospective module uses reconstruction error and sparse latent representations as intrinsic rewards to encourage coarse and fine-grained exploration. A confidence-based mechanism switches between the memory policy and the exploration policy. The authors also provide theoretical claims on policy optimality preservation and ensemble improvement. Experiments across Atari, MuJoCo, and several navigation environments report substantial improvements over existing baselines, particularly on Montezuma's Revenge.

Critical strengths
  1. A relevant approach to adaptive exploration-exploitation coordination.

The paper addresses an important problem in RL: how to adapt exploration and exploitation according to accumulated experience rather than relying solely on predefined schedules. The combination of retrospective memory and prospective exploration provides a reasonable framework for studying this problem.

  1. Explicit consideration of both successful and failed experiences.

The retrospective module incorporates high-return trajectories and failure-related state-action pairs through separate memory banks. Using a reflection network to assess the reliability of memory-based actions is an interesting design choice, particularly for environments where unsuccessful actions can be costly.

  1. Extensive empirical evaluation across different environments.

The paper reports experiments across a broad collection of discrete and continuous control tasks. The improvements on Montezuma's Revenge are particularly notable, with MemoRL+PPO achieving a reported return of 6652 compared with 440 for PPO under the evaluated setting. The results suggest that the framework can substantially improve performance in challenging exploration tasks.

  1. Useful component-level analyses.

The experiments on Cliff Walking, Dark Chamber, and Four Rooms provide intuitive illustrations of the proposed mechanisms. Table 2 also evaluates the contributions of the exploration, exploitation, and coordination components, offering useful evidence that their combination can outperform individual modules.

Critical weaknesses
  1. The theoretical guarantee for ensemble improvement is questionable.

In Eq. (5), the confidence-dependent indicator is action-dependent, so the resulting mixture of policies does not necessarily form a normalized probability distribution. This also differs from Algorithm 1, which evaluates a single memory action before switching policies. Furthermore, Theorem 2 assumes that the memory policy is greedy with respect to the original policy's Q-function, whereas the actual memory policy is trained through behavior cloning (Eq. 1). Therefore, the theorem does not directly establish improvement for the implemented algorithm.

  1. The optimality-preservation guarantee relies on a restrictive assumption.

Theorem 1 requires the intrinsic reward to be bounded by the minimum optimal-action gap, which is generally unknown and may approach zero in continuous action spaces. This makes Assumption 1 difficult to satisfy or verify in practice. Additionally, the proof in Appendix B.1 uses a strict inequality that is not guaranteed by the non-strict assumption. The practical significance and precise scope of this theoretical guarantee remain unclear.

  1. Insufficient justification for the fine-grained exploration mechanism.

The prospective module uses reconstruction error and L1 regularization on latent features as intrinsic rewards (Eq. 3). However, the paper does not establish why latent sparsity should reliably distinguish visually similar states with different decision-making importance. Figures 5 and 6 demonstrate improved exploration coverage but do not directly validate this central claim. More targeted evidence connecting the proposed sparsity measure to state novelty or action relevance is needed.

  1. Limited distinction between retrospective learning and existing imitation-based methods.

The Digest Network uses standard behavior cloning on high-return trajectories (Eq. 1), while the Reflection Network performs binary classification on successful and failure-related state-action pairs (Eq. 2). These operations do not explicitly identify failure causes or infer corrective actions. In particular, actions immediately preceding failure are not necessarily responsible for it. The methodological distinction from existing self-imitation and failure-aware learning approaches is therefore insufficiently established.

  1. The ablations do not fully isolate the proposed contributions.

Table 2 evaluates the removal of major modules but provides limited insight into their individual mechanisms. Comparisons against success-only imitation, failure-only reflection, and simpler policy-switching strategies would better establish the contribution of retrospective learning and adaptive coordination. Similarly, separating reconstruction-based novelty from the latent sparsity penalty would clarify the benefit of fine-grained exploration.

  1. Experimental comparisons and efficiency claims require clarification.

Section 4.2 reports a 41.8% efficiency improvement, although MemoRL requires approximately 1.43 times the training time of RND for the same environment steps. The distinction between sample efficiency and wall-clock efficiency should be clarified. Moreover, Tables 3 and 4 lack sufficient details on matched computational budgets and evaluation protocols, making comparisons with independently reported results difficult to assess. The unusually large Frostbite result for Size-512 in Table 2 also warrants verification.

Questions for authors
  1. Ensemble policy and theoretical guarantee: How is Eq. (5) guaranteed to define a normalized probability distribution when the confidence indicator depends on the action? Could the authors clarify the discrepancy between Eq. (5) and Algorithm 1, and explain how Theorem 2 applies to the implemented memory policy?

  2. Optimality preservation: How is Assumption 1 justified or verified during training? In particular, how can the guarantee remain informative in continuous action spaces where the minimum optimal-action gap may approach zero?

  3. Fine-grained state discrimination: What evidence shows that latent sparsity specifically captures differences in decision-making importance between visually similar states, rather than simply reflecting representation magnitude or reconstruction difficulty?

  4. Retrospective learning: How does the Reflection Network distinguish genuinely harmful decisions from actions that merely precede a failure? Have the authors compared the method with success-only imitation and failure-only reflection under matched conditions?

  5. Experimental attribution and efficiency: How does the learned confidence-based policy selection compare with simpler switching strategies? Could the authors also clarify whether the reported 41.8% efficiency improvement accounts for the additional wall-clock training cost?

Reviewer_qnKTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemoRL, which coordinates exploration and exploitation through retrospective and prospective modules. The retrospective module learns a memory policy from advantageous trajectories and uses failure examples to assess the reliability of memory actions. The prospective module combines state reconstruction error and sparse representations to construct an intrinsic reward for exploration. A reliability score determines whether to follow the memory policy or the base policy. The paper provides conditional theoretical analysis and evaluates the framework on Atari, MuJoCo, and diagnostic environments, reporting improvements over several base algorithms.

Critical strengths

The motivation is clear, and combining successful and failed experience is an interesting idea. Beyond reusing high-return trajectories, MemoRL uses failure experience to assess the reliability of memory actions and decide when to exploit past experience or continue exploring. This offers a reasonable and potentially useful approach to exploration–exploitation coordination in sparse-reward tasks. The evaluation is relatively broad, and the main results show practical promise. The paper evaluates the framework on Atari and MuJoCo and reports integration with PPO, RND, and RPO. The improvement on Montezuma’s Revenge is substantial, and the module-level ablations in Table 2 support the benefit of combining exploration and exploitation. Using five seeds for the main-text Atari experiments also adds value to the evaluation.

Critical weaknesses

The existing ablations do not fully separate the independent benefits of the two central components. The experiments evaluate the retrospective module as a whole, include heatmap comparisons without the F-Discriminator, and examine modules and feature dimensions. However, they lack an independent quantitative control that retains successful-experience distillation while omitting only failure-example training, as well as multi-seed return statistics for removing the fine-grained term on the main Atari tasks. Adding these two controlled comparisons on a few representative tasks, with means and variability under the same interaction budget, would more directly establish the separate contributions of failure reflection and fine-grained exploration. The efficiency evaluation provides relative results, but the reporting of actual running times remains incomplete. The paper reports relative time savings to a common reward threshold and states that training takes approximately 1.43 times as long as RND at a fixed step budget. However, it does not provide per-task absolute times, variability across seeds, or specific hardware models. On a few representative tasks, I suggest reporting actual running-time statistics under a consistent hardware setup, or return-versus-wall-clock-time curves. In addition, the text refers to 12 environments while the efficiency table lists five tasks; the five reported percentages average to exactly 41.8%, so the task set underlying the reported mean should be specified.

Questions for authors

Could the authors provide multi-seed quantitative controls that omit only failure-example training or remove the fine-grained term, to measure the separate contributions of these components? Does the reported 41.8% average time saving refer to the five tasks in the efficiency table or the twelve environments mentioned in the text? Could the authors provide the hardware models, absolute times, and variability across seeds for representative tasks? Table 5 reports ReliabilityThreshold=0.85 and Lambda=0.95. Does Lambda refer to the sparsity coefficient in Eq. (3) or the GAE parameter? Also, what action-selection rule is used for “w/o Trade-off” in Table 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.