acceptodds
Under review as a conference paper at ICLR 2027

Solving Sparse-Reward Environments using Meta-Gradients

Abstract

Exploration is essential in reinforcement learning since agents rely on trial and error to learn an optimal policy. However, when rewards are sparse, naive exploration strategies, like noise injection, are often insufficient. Intrinsic rewards can provide principled guidance for exploration by combining them with extrinsic rewards to optimize a policy or using them to train subpolicies for hierarchical learning. However, the former approach induces suboptimality as it changes the performance objective, while the latter is sample inefficient and also suboptimal. To address these challenges, we propose an algorithm, intrinsic reward policy optimization (IRPO), that uses intrinsic rewards within a meta-gradient framework to drive exploration without changing the performance objective. We achieve this by updating a shared meta-policy such that one of the subpolicies (derived from the meta-policy using intrinsic rewards) optimizes extrinsic rewards. We demonstrate that IRPO outperforms relevant baselines in sparse-reward environments with similar wall-clock time to standard policy gradient methods, and justify IRPO design choices through ablation studies. We also prove that IRPO converges to an -approximate first-order stationary point.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, intrinsic rewards, sparse-rewards, meta-gradients

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fy8yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed intrinsic reward policy optimization (IRPO), a meta reinforcement learning algorithm to encourage exploration. In IRPO, they first train ALLO intrinsic rewards, each rewarding agents to proceed different directions. They maintain multiple sub-policies optimized for different ALLO intrinsic reward to ensure the exploration, in the inner loop. In the outer-loop, the initial policy is updated to maximize the environmental reward. Theoretically, they have provided the evidence that within this framework, sub-policies, updated for intrinsic rewards will converge to the stationary points. In experiment, they validated the performance of IRPO in several goal-reaching environments.

Critical strengths
  • While meta-RL is typically used for adaptation across different tasks, this paper applies it to exploration in sparse-reward environments, which I found refreshing. The separation of intrinsic-reward-based inner-loop and extrinsic-reward-based outer-loop provides a clear and intuitive framework.
  • The theoretical guarantee justifies the usage of non environmental rewards for the sub-policy updates
  • The diversity of the baselines, from vanilla on-policy methods to other intrinsic bonuses, strengthened the experimental results.
Critical weaknesses

The main weakness is the assumption that initial states can be sampled uniformly from the state space when pretraining ALLO intrinsic rewards. While this may be possible in the relatively small simulated environments used in the paper, it would be difficult in real-world environments or even in challenging simulated environments with sparse reward such as Montezuma's Revenge. Since IRPO relies on pretrained and fixed ALLO intrinsic rewards, violating this assumption could substantially degrade its performance. This may limit the applicability of the proposed method. Related to this, the paper also lacks the investigation on the how the coverage of the data for ALLO training affects the resulting RL training.

Questions for authors
  • Can authors provide some results on the relationship between the coverage of the data for ALLO training and the RL performance?
  • Is it possible to train ALLO intrinsic reward online manner to overcome the limitation posed in the weakness?
10 comments
AuthorsOpenReview ↗

Author Reponse

We thank the reviewer for finding our work refreshing and seeing value in our theoretical guarantee and diversity of baselines. We also appreciate the concerns and suggested improvements.

1. [Edited November 16] The main weakness is the assumption that initial states can be sampled uniformly from the state space when pretraining ALLO intrinsic rewards.

  • Our revision now provides full IRPO-ALLO results using ALLO pretrained with the same fixed initial states used for policy training. See Table 1 (peak eval return) and Figure 6 (learning curves).
  • IRPO-ALLO still outperforms all baselines except in PointMaze-v2, where it ranks second (0.99 vs. 0.91 return).
  • A summary of the results comparing IRPO-ALLO (fixed initial state for ALLO) and IRPO-ALLO (random initial state for ALLO) is given in the table below. | Environment | IRPO-ALLO (uniform) | New IRPO-ALLO (fixed) | Δ Mean | | :--- | ---: | ---: | ---: | | Maze-v1 | 0.83 ± 0.00 | 0.83 ± 0.00 | 0.00 | | Maze-v2 | 0.83 ± 0.00 | 0.83 ± 0.00 | 0.00 | | FourRooms | 0.88 ± 0.00 | 0.88 ± 0.00 | 0.00 | | PointMaze-v1 | 0.85 ± 0.02 | 0.78 ± 0.02 | −0.07 | | PointMaze-v2 | 0.85 ± 0.01 | 0.68 ± 0.17 | −0.17 | | FetchReach | 0.86 ± 0.02 | 0.97 ± 0.00 | +0.11 | | AntMaze-v1 | 0.99 ± 0.00 | 0.91 ± 0.07 | −0.08 | | AntMaze-v2 | 0.97 ± 0.01 | 0.93 ± 0.03 | −0.04 | | AntMaze-v3 | 0.97 ± 0.01 | 0.93 ± 0.05 | −0.04 |
  • PPO-ALLO (fixed) and HRL-ALLO (fixed) are still running, and we expect them to finish before Wednesday, November 18. They were not competitive with randomly initialized ALLO, so we do not expect them to be competitive with fixed initial state ALLO. We will update the revision then.

2. [Edited November 16] Can the authors provide results on the relationship between the coverage of the data for ALLO training and the RL performance?

  • Our revision now provides a new ablation (see Figure 7i) evaluating the performance of IRPO-ALLO while varying the number of samples used to pretrain ALLO.
  • We see that the performance of IRPO-ALLO decreases slightly as the number of samples used to pretrain ALLO decreases.
ALLO pretraining samplesMean peak return
1000.62
1k0.62
10k0.70
50k0.70
100k0.70
200k0.78
  • Note that 200k samples is at most 0.02% of the total samples used to train all algorithms

3. Is it possible to train ALLO intrinsic reward online manner to overcome the limitation posed in the weakness?

  • This is a good idea. While it is possible, we were unable to generate the data within the deadline (we focused on generating data where ALLO uses the same fixed initial position as all other baselines). Our revision mentions this as a future direction.

Thank you for your rebuttal and additional experiments. The new experiments, which pretrain ALLO using random walks from fixed initial states while largely maintaining performance, alleviate my original concern that the algorithm requires the ability to reset the environment to arbitrary states.

However, my broader concern remains. In large, hard-exploration environments (often sparse reward), random walks from the initial state covers only a small portion of the state space. The performance degradation observed with fewer ALLO pretraining samples suggests that insufficient training data can affect performance, and limited state-space coverage may pose an even greater challenge, particularly in regions not visited during pretraining. (It would be helpful to know how much of the reachable state space is covered by random walks in environments such as Maze and PointMaze.)

This is why I raised online ALLO training as a question that could affect my assessment. I understand that evaluating this possibility within the rebuttal period was difficult. Nevertheless, I believe that the dependence on the coverage of the ALLO pretraining data remains a limitation of the current method. Therefore, it is difficult to raise my score at this stage.

AuthorsOpenReview ↗

New results on online training

Thank you for your thoughtful response. We are glad we were able to address your concerns regarding reset requirements.

We now better understand your concern regarding how much coverage of the state space is required for ALLO to produce useful intrinsics. We agree that our current method may struggle to handle environments where "sufficient" coverage of the state space (via random walk for ALLO) is not possible within a reasonable amount of samples.

We were actually able to generate results where ALLO is jointly trained (online) with IRPO policies (in FourRooms PointMaze, and FetchReach). We added these results to Section 4.3 of the revision.

  • We see that joint IRPO-ALLO achieves similar converged performance to IRPO-ALLO (200k pretrained ALLO, which generally provides good coverage). While joint IRPO-ALLO can require more samples than IRPO-ALLO, it still outperforms baselines.
  • We summarize peak eval return below. Learning curves can be found in Figure 9 of the revision. | Environment | IRPO-ALLO | Joint IRPO-ALLO | |---|---:|---:| | FourRooms | 0.8773 | 0.8529 | | PointMaze-v1 | 0.7801 | 0.7799 | | FetchReach | 0.9679 | 0.9378 |
  • We have provided the implementation detail of the joint training of IRPO in Appendix C.2.
  • And thank you for the idea of online joint training.
Reviewer_jya3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces intrinsic reward policy optimization (IRPO), a meta-gradient method which utilizes intrinsic rewards without modifying the optimality of the meta-objective. Intrinsic motivation methods add an exploration incentive to the reward function, shifting the optimal policy away from that of the extrinsic (environment) reward function. IRPO addresses this by training individual sub-policies with intrinsic rewards, then combining their individual updates into a meta-update with respect to the extrinsic reward function objective, and then updates the meta-policy with a trust region update in the combined meta-gradient direction. The individual updates are combined with a weighted softmax according to the performance of the corresponding sub-policy on the extrinsic reward objective. By annealing the temperature of this softmax to 0, the weighting more heavily prioritizes alignment to the extrinsic reward, and the combined update produces a meta-policy from which a single subpolicy produces the optimal policy under the extrinsic reward. This behavior allows the paper to provide a proof of convergence guarantees for IRPO.

The paper further validates IRPO empirically on 9 environments against 7 baselines including PPO, MAML, hierarchical methods, and intrinsic motivation methods. IRPO matches or outperforms all of these baselines across environments in 10 independent runs. Furthermore, the subpolicy updates can be parallelized, resulting in no significant increase in training time. The appendix has a thorough analysis of how compute time and VRAM requirements grow with the number of intrinsic rewards and subpolicy updates, though in practice the method trains in a similar amount of time as PPO.

Critical strengths

The main strengths of the paper are it's empirical thoroughness, and novelty. The paper compares against 7 baselines in 9 environments each with 10 seeds, and also has ablates several design choices and includes additional analyses of compute usage and in the appendix. The method is also interesting and addresses a clear limitation of intrinsic rewards: they modify the optimal policy for a task. I am not aware of any work that takes the approach described here, and was unable to find anything too similar in a brief literature search

The paper has a clear presentation that walks the reader through the intuition behind the method sufficiently well, though some questions remain. In particular, the intuition behind the convergence proof is quite clear, which is extremely valuable because the method itself is quite complex.

Critical weaknesses

The main weakness of the paper in my view is the quality of the environments and baselines. There are exploration methods which can address significantly more challenging settings than those studied here (such as Elliptical Episodic Bonuses in MiniHack, or Coin Flip Networks in Montezuma's Revenge and 42x42 gridworlds). The fact that IRPO achieves over an 83% success rate in every environment also suggests that it would be beneficial to test on harder domains. However, I believe the chosen environments and methods are sufficient to justify the claims in the paper, and the work provides a solid foundation for future research as is.

Some additional points you may wish to address, but which will not impact my score:

  • The related works can be hard to follow because you reference papers without method names. For instance, I was wondering how your method compared to LIRPG, but I did not remember the author name, so it took some effort to locate it in the related works.
  • The confidence intervals do not appear to be accurate, I don't believe 10 samples are sufficient to assume normality and use symmetric confidence intervals. These should be calculated as 95% bootstrap confidence intervals and reported as two values.
  • The paper does not appear to report p values for any comparisons, which can be valuable in addition to confidence intervals. That being said, I expect many of the main results are statistically significant based on the reported values. I recommend adding the results of a Mann-Whitney U-test against all baselines in Table 1 to the appendix.
Questions for authors
  1. Can you provide some explanation for how IRPO is as sample efficient as other standard RL methods despite training K times more policies on non-extrinsic rewards? I assume that IRPO would be significantly less sample efficient that PPO in dense reward environments where PPO learns successfully.
  2. Was ALLO chosen specifically because it can produce multiple intrinsic rewards? Have you tried using other standard intrinsic rewards (ICM, RND, E3B, etc)?

I'm also curious about the following but don't expect the responses to impact my review: 3. Do you see a connection to ensemble methods for exploration, which train many different policies and calculate an intrinsic reward from them (https://proceedings.mlr.press/v97/pathak19a.html, https://proceedings.neurips.cc/paper_files/paper/2023/file/2a4310c4fd24bd336aa2f64f93cb5d39-Paper-Conference.pdf)? 4. Why are there 2K critics? Does each sub-policy have an intrinsic and extrinsic critic?

3 comments
AuthorsOpenReview ↗

Author Reponse

We thank the reviewer for finding our work novel, clearly presented, and our empirical studies thorough, and for providing suggestions for further improvement.

1. Related works can be hard to follow because you reference papers without method names.

  • We agree. Our revised ``Related Work'' section now references papers using method names.

2. [Confidence intervals] should be calculated as 95% bootstrap confidence intervals and reported as two values.

  • Thank you for the suggestion. Our revision updates Figure 6 (learning curves) and Figure 7 (ablation learning curves) to show 95% bootstrap confidence intervals.

3. I recommend adding the results of a Mann-Whitney U-test against all baselines in Table 1 to the appendix.

  • Thank you for the suggestion. Our revision includes the requested table (Table 5 in Appendix D) and a discussion of results in Appendix D.

4. Can you provide some explanation for how IRPO is as sample efficient as other standard RL methods despite training K times more policies on non-extrinsic rewards? I assume that IRPO would be significantly less sample efficient that PPO in dense reward environments where PPO learns successfully.

  • One explanation is that since ALLO provides (at least some) useful intrinsic rewards for the environments considered, those corresponding subpolicies will collect rewards sooner than the (unguided) exploration seen by PPO. While early on, some samples will be wasted (by subpolicies following not useful intrinsics), once some rewards are received by a subpolicy, the IRPO gradient then ensures the meta-policy (and therefore future subpolicies) is updated in a direction that will continue to receive those rewards. After enough meta-policy updates, even less useful intrinsics will then still likely produce some useful rewards.
  • In dense reward settings, the useful subpolicy (or subpolicies) just collects rewards even sooner, leading to similar sample efficiency gains relative to PPO.
  • It would be interesting to more thoroughly study this hypothesis, which we hope to do in future work. We do acknowledge this benefit will likely disappear in extremely dense reward settings where exploration is not a challenge.

5. Was ALLO chosen specifically because it can produce multiple intrinsic rewards? Have you tried using other standard intrinsic rewards (ICM, RND, E3B, etc)?

  • ALLO was chosen because it has been shown to generate good intrinsic rewards for navigation (Gomez et al. 2023). In principle, IRPO should work fine with a single intrinsic reward (since the meta-policy will just ``follow'' that subpolicy), as long as that reward is useful for exploration.
  • We have not tried the suggested intrinsic rewards (ICM, RND, E3B), but our ablation in Figure 7 shows results for IRPO with DRND (successor of RND) and LIRPG intrinsic rewards, as well as random intrinsics. We see that IRPO-ALLO generally does best, though IRPO-DRND does well in FetchReach (but not other environments). We plan to more rigorously study the use of other intrinsics in future work.

6. Do you see a connection to ensemble methods for exploration, which train many different policies and calculate an intrinsic reward from them?

  • This is an interesting question. We believe the suggested works fall in the uncertainty-based category of related work. They can actually be used with IRPO (replacing ALLO). We plan to study this in future work. Our revision adds these to the related work section.

7. Why are there 2K critics? Does each sub-policy have an intrinsic and extrinsic critic?

  • Yes, each subpolicy has an intrinsic critic (used to update the subpolicy) and an extrinsic critic (used to update the meta-policy). See Vϕ~k(j)V_{\tilde{\phi}_k^{(j)}} (intrinsic critic) and Vϕk(j)V_{{\phi}_k^{(j)}} (extrinsic critic) in Figure 2.

Thank you for taking the time to answer my questions and implement my suggestions. My concern regarding more complex environments remains, but my other comments and questions have been sufficiently addressed. Regardless, I feel this work should be accepted.

Thanks for your response.

Regarding more complex environments, we mistakenly did not mention that we are currently generating data for IRPO-ALLO and PPO in MiniGrid Door Key and MiniGrid Key Corridor (environments shown https://drive.google.com/file/d/1alfp6k5Ih3XbykvgWf-rJkk8FBv2R_KU/view?usp=sharing).

  • We expect to have complete data by Monday November 16 and will update our response and revision then.
  • Our preliminary results (over 3 seeds), shown below, show that IRPO-ALLO can solve both environments while PPO fails.
  • We plan to include these results (over 10 seeds) in the final paper.
EnvironmentIRPO-ALLOPPO
Door Key0.914 ± 0.0000.032 ± 0.062
Key Corridor0.860 ± 0.0000.002 ± 0.007

We agree there is more interesting work to be done exploring (significantly) more complex environments though, and plan to do so in future work.

Reviewer_sV5bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IRPO, a meta-gradient method for exploration in sparse-reward environments. The goal is to benefit from intrinsic rewards without changing the original objective, as reward augmentation does. Several subpolicies derived from a shared meta-policy explore using different intrinsic rewards proposed by a previous work, and the meta-policy is updated so that at least one of these subpolicies performs well on the extrinsic reward. The authors prove convergence to a stationary point and show that IRPO outperforms meta-learning baselines on a set of discrete and continuous navigation tasks.

Critical strengths
  1. The paper is clearly written and well presented. The idea of exploring through intrinsic rewards in an inner loop while the outer loop optimizes only the extrinsic reward is interesting, and the toy illustrations (Figs. 3 and 9) help a lot.
  2. The evaluation is careful: 10 seeds with confidence intervals, ablations on temperature, number of inner updates, meta-gradient vs. importance sampling, and the choice of intrinsic reward, plus a compute analysis showing wall-clock time similar to PPO.
Critical weaknesses
  1. Important baselines are missing. EIPO (Chen et al. 2022), which has very similar setting and is cited but not compared. Some baselines also look poorly tuned: MAML gets always −1 on all AntMazes and LIRPG is near zero almost everywhere.
  2. The scalability of backpropagating through N inner updates to larger networks is unclear ( see in Fig. 10). I would be curious about the computation time.
  3. ALLO is trained in the same tasks, giving unfair advantages due to its reset mechanism, which allows reset to random position in the maze.
Questions for authors
  1. How does IRPO perform when the intrinsic rewards are learned without uniform resets or access to the agent's coordinates?
  2. Can you compare against EIPO, which targets the same suboptimality issue?
  3. How does IRPO perform on harder exploration benchmarks beyond navigation, e.g. tasks with keys/doors (MiniGrid) or pixel observations?
  4. Would the intrinsic reward learned by ALLO actally transfers?
2 comments
AuthorsOpenReview ↗

Author Response

We are glad the reviewer found our work well presented, our idea interesting, and our evaluation careful. We appreciate your comments, which we respond to below.

1. Missing baselines (EIPO).

  • Our revision adds results for EIPO to Table 1 (peak evaluation return), Figure 6 (learning curves), and Figure 7 (ablation). Hyperparameters were tuned using the same Bayesian hyperparameter sweep used for other algorithms.
  • We see that IRPO outperforms EIPO. Our revision (Section 4.1) adds that ``EIPO-ALLO shows modest performance across environments but is outperformed by IRPO-ALLO for all environments. This is likely because EIPO is an uncertainty-based reward augmentation method and therefore suffers from suboptimality. Note that EIPO-ALLO generally outperforms PPO-ALLO (except in AntMaze).''

2. Some baselines look poorly tuned (MAML, LIRPG).

  • We tuned hyperparameters for each algorithm, including MAML and LIRPG, using the same process (a Bayesian hyperparameter sweep through WandB). Appendix A.C.1 details our process and hyperparameters.
  • MAML struggles due to a mismatch between its intended purpose---few-shot adaptation---and our problem of exploration in sparse-reward environments.
  • LIRPG requires extrinsic rewards to learn useful intrinsic rewards, leading to its poor performance in environments where extrinsic rewards are sparse.

3. The scalability of backpropagating through N inner updates to larger networks is unclear (see Fig. 10). I would be curious about the computation time.

  • Our revision updates Figure 10 to plot neural network size on the x-axis and wall-clock computation time per-iteration (or peak allocated VRAM) on the y-axis. Hopefully this more clearly shows the relationship between neural network size and computation time/GPU memory usage.
  • The corresponding text also more clearly states that ``Figure 10 shows that both per-iteration wall-clock time and GPU memory usage increase sublinearly with the number of neural network parameters LL over the measured range, and approximately linearly with KK and NN.''

4. [Edited November 17] It is unfair to use random resets when training ALLO. How does IRPO perform when the intrinsic rewards are learned without uniform resets?

  • Our revision now provides full results using ALLO pretrained with the same fixed initial states used for policy training (for IRPO-ALLO and all ALLO baselines). See Table 1 (peak eval return) and Figure 6 (learning curves).
  • IRPO-ALLO still outperforms all baselines except in PointMaze-v2, where it ranks second (0.99 vs. 0.91 return).
  • A summary of the results comparing IRPO-ALLO (fixed initial state for ALLO) and IRPO-ALLO (random initial state for ALLO) is given in the table below. | Environment | IRPO-ALLO (uniform) | New IRPO-ALLO (fixed) | Δ Mean | | :--- | ---: | ---: | ---: | | Maze-v1 | 0.83 ± 0.00 | 0.83 ± 0.00 | 0.00 | | Maze-v2 | 0.83 ± 0.00 | 0.83 ± 0.00 | 0.00 | | FourRooms | 0.88 ± 0.00 | 0.88 ± 0.00 | 0.00 | | PointMaze-v1 | 0.85 ± 0.02 | 0.78 ± 0.02 | −0.07 | | PointMaze-v2 | 0.85 ± 0.01 | 0.68 ± 0.17 | −0.17 | | FetchReach | 0.86 ± 0.02 | 0.97 ± 0.00 | +0.11 | | AntMaze-v1 | 0.99 ± 0.00 | 0.91 ± 0.07 | −0.08 | | AntMaze-v2 | 0.97 ± 0.01 | 0.93 ± 0.03 | −0.04 | | AntMaze-v3 | 0.97 ± 0.01 | 0.93 ± 0.05 | −0.04 |

5. How does IRPO perform when the intrinsic rewards are learned without access to the agent's coordinates?

  • ALLO (as originally implemented in (Gomez et al. 2023) and implemented in our work) requires agent coordinates to learn intrinsic rewards that are useful for spatial navigation.
  • We also want to add that all environments tested include the agent's coordinates as part of the state---removing these coordinates will likely cause all methods to fail.

6. [Edited November 16] How does IRPO perform on harder exploration benchmarks beyond navigation, e.g., tasks with keys/doors (MiniGrid) or pixel observations?

  • Our revision now includes complete experiments comparing IRPO-ALLO (fixed initial state ALLO), PPO, PPO-DRND, and PPO-LIRPG on Door Key and Key Corridor over 10 random seeds. See Figure 8 and corresponding discussion in Section 4.3.
  • IRPO-ALLO achieves strong performance (0.914 in DoorKey and 0.86 in KeyCorridor), while the baseline algorithms fail (~0).

7. Would the intrinsic rewards learned by ALLO transfer?

I would appreciate the authors' efforts addressing many of my concerns. I now have a better understanding of the work. I think the experiments the authors carried out are more solid than the original version. Yet due to the impact concern, I will keep my current score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.