- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a lightweight and easy-to-implement exploration approach that can be used with both model-free and model-based RL methods. The proposed approach injects perturbations into the reward function based on the TD error and the estimated state value relative to the batch of states collected to update the value model. The sign of the perturbation is determined to direct the agent toward underexplored regions that may contain promising states but currently have low target values. The authors provide extensive experiments in both single-agent and multi-agent environments and demonstrate improved performance compared to the baselines.
Critical strengths
The paper proposes a method to address the well-known challenge of exploration in RL. The authors provide extensive experiments across multiple domains and compare their method with recent baselines proposed to address the same challenge.
Critical weaknesses
My main concern is the motivation behind BiRN's claimed superiority over SrRN in sparse-reward environments. The authors argue that BiRN improves exploration by determining the direction of reward perturbations based on TD errors and the average bootstrap target of the batch. However, it is unclear why this mechanism should be particularly effective in sparse-reward settings. In environments where rewards are only given upon reaching the goal, all observed rewards may remain zero until the first successful trajectory is found. With zero-initialized value estimates, this would lead to zero TD errors, meaning that BiRN would not inject any reward noise. Even with nonzero initial value estimates, the TD errors and target differences may mainly reflect random initialization or estimation errors rather than useful information about promising unexplored states.
Therefore, it is unclear why using these values to determine the direction of reward perturbations should lead to better exploration than the direction-blind noise used in SrRN or even the RND method. The authors should provide a clearer theoretical explanation or empirical evidence supporting this mechanism, particularly by investigating how BiRN encourages exploration before the agent discovers its first reward in genuinely sparse-reward environments.
Questions for authors
1- Figure 1 lacks sufficient explanation. It is unclear which environment or experimental setting the figure represents, making its interpretation difficult. The authors should provide a more detailed explanation of the figure and clarify how it illustrates the proposed method.
2- The illustrative gridworld environment appears to be specifically designed to favor the proposed exploration method. Moreover, the additional results reported in the appendix show that count-based exploration achieves greater state coverage than the proposed method during early training and reaches nearly identical returns after 50K training episodes. These findings raise questions about the claimed exploration advantages of the proposed method.
3- The evaluation on Atari games is limited to only four environments. It would be valuable to evaluate the proposed method on more challenging sparse-reward exploration benchmarks, such as Montezuma's Revenge and Pitfall, to better assess its effectiveness in environments requiring substantial exploration.
4 comments
Response to the questions
Q1: Figure 1 lacks sufficient explanation. It is unclear which environment or experimental setting the figure represents, making its interpretation difficult. The authors should provide a more detailed explanation of the figure and clarify how it illustrates the proposed method.
Thank you for pointing this out. Figure 1 is intended as a conceptual illustration rather than a visualization of any specific environment or experimental setting. The figure illustrates the two complementary effects of BiRN: negative perturbations reduce the height of relatively high-reward regions that may attract premature exploitation, while positive perturbations strengthen alternative regions with relatively lower targets, provided that their TD errors are positive.
We agree that the original figure caption does not sufficiently explain these elements. We have revised the accompanying description to clarify the conceptual nature of the figure and the role of each directional perturbation.
Q2: The illustrative gridworld environment appears to be specifically designed to favor the proposed exploration method. Moreover, the additional results reported in the appendix show that count-based exploration achieves greater state coverage than the proposed method during early training and reaches nearly identical returns after 50K training episodes. These findings raise questions about the claimed exploration advantages of the proposed method.
The gridworld is designed as a controlled setting to examine the premature-reward-trap scenario targeted by BiRN, where an easily accessible suboptimal reward competes with a more distant, higher reward. This setting makes it possible to directly examine whether the proposed directional perturbation helps overcome premature exploitation.
BiRN's exploration advantage lies in reducing premature exploitation of suboptimal rewards, rather than maximizing overall state coverage. Although PPO-Count achieves slightly higher state coverage during early training (97.07% vs. 94.67%) and a similar final return, its broader coverage fails to translate into faster progress toward the higher-reward solution: it visits more than twice as often as BiRN (142 vs. 62 visits) and achieves lower returns at intermediate training stages. Thus, BiRN reduces repeated visits to suboptimal reward regions and accelerates progress toward higher-reward solutions.
Importantly, BiRN's applicability is not limited to this illustrative gridworld. It provides a lightweight and readily integrable exploration mechanism by reusing TD errors and bootstrap targets already available in Bellman-style RL algorithms, without requiring additional trainable components or modifications to network architectures. Its evaluation across discrete and continuous control, sparse-reward tasks, and multi-agent environments further demonstrates its applicability to diverse RL settings.
Q3: The evaluation on Atari games is limited to only four environments. It would be valuable to evaluate the proposed method on more challenging sparse-reward exploration benchmarks, such as Montezuma's Revenge and Pitfall, to better assess its effectiveness in environments requiring substantial exploration.
We appreciate the suggestion. Our evaluation aims to examine BiRN across diverse RL environments, including Atari games, continuous-control tasks, sparse-reward navigation, and multi-agent environments, rather than focusing exclusively on expanding the number of Atari games. Nevertheless, following the reviewer's suggestion, we extended our Atari evaluation by adding Pitfall and Montezuma's Revenge. Due to the limited time, each run was trained for 40M environment steps.
Pitfall. PPO+BiRN achieved the highest mean final game score of −0.342, compared with −0.726 for PPO. However, PPO, PPO+RND, and PPO+BiRN did not obtain positive extrinsic rewards within this training budget. Thus, BiRN's higher score primarily reflects fewer penalties rather than successful discovery of positive-reward trajectories.
| Method | Final game score ↑ |
|---|---|
| PPO | −0.726 ± 0.315 |
| PPO + NoisyNet | −0.607 ± 1.144 |
| PPO + RLE | −0.948 ± 0.286 |
| PPO + RND | −3.852 ± 0.418 |
| PPO + BiRN | −0.342 ± 0.081 |
Montezuma's Revenge. Under the same 40M-step budget, PPO, PPO+RND, and PPO+BiRN did not exhibit sustained reward growth or reliably discover rewarding states. Although occasional nonzero scores were observed, none of the methods achieved consistent progress.
These additional experiments extend our evaluation to two challenging sparse-reward Atari games. While BiRN achieves a higher final score on Pitfall and shows occasional positive-reward discovery on Montezuma's Revenge, reliably discovering and accumulating rewards remains challenging under the 40M-step budget. This is consistent with BiRN's role in regulating existing reward-driven learning signals, rather than providing reward-free exploration.
Response to Authors’ Comments
Thank you for providing explanations in response to my comments. I was expecting more theoretical justification for why your approach would be beneficial in sparse-reward environments, where the agent might not encounter a reward until much later in training, especially when the goal is difficult to reach. This is also consistent with what the authors showed empirically in the Montezuma's Revenge environment.
Overall, I maintain my positive score, but I was not satisfied with your justification regarding RND and the comparison between BiRN and SrRN. Since your method aims to improve exploration, it should either outperform the other baselines or at least demonstrate reasonable performance in long-horizon, sparse-reward environments.
Thank you for encouraging us to provide a deeper theoretical analysis of BiRN under sparse rewards.
Our key observation is that sparse rewards make successful feedback both rare and concentrated, which can cause a small number of behaviors to dominate the learning signal. Consider a state with two actions and , whose subsequent trajectories reach the same terminal goal with probabilities and . If independent trajectories are sampled after each action, let be the probability of observing at least one success for action . Conditional on observing any success, the probability that only one action produces a success is . This approaches when . Thus, even when two actions are comparably promising, a success, when observed under sparse feedback, is likely to come from only one action. The probability of observing any success, , also decreases as the goal becomes harder to reach.
Proposition 1 and BiRN's perturbation rule show how directional perturbation reduces the expected target gap when sparse reward feedback is concentrated on a few behaviors. Consider transitions from this decision state, with and . Under the perturbation analyzed in Proposition 1, , whereas SrRN leaves the expected gap unchanged. Thus, once a rare success is reflected in a positive TD signal for while , BiRN reduces the relative reinforcement of . If value information subsequently propagates to a transition following , giving it despite zero immediate reward, BiRN also strengthens this alternative transition. The expected gap reduction therefore increases from to . Under a local two-action softmax policy-gradient update using TD errors as fixed advantages, this smaller gap reduces the update favoring and increases the probability of trying to first order.
This analysis also clarifies the distinct roles and limitations of BiRN relative to RND and SrRN. RND provides reward-independent exploration signals and can therefore encourage exploration before any extrinsic reward is observed. BiRN, in contrast, relies on reward-derived TD signals and remains inactive when both observed rewards and value estimates are zero. However, once sparse reward feedback begins to shape the learning targets, BiRN counteracts the disproportionate influence of early rewarding transitions. Unlike SrRN, whose zero-mean perturbations leave the expected target gap unchanged, BiRN directionally adjusts the relative learning signals and can further strengthen alternative transitions as they acquire positive TD errors. In our 40M-step Montezuma's Revenge runs, PPO and PPO+BiRN occasionally obtained positive scores, but neither showed sustained reward growth, suggesting that reliably discovering and reinforcing rewarding trajectories remained a bottleneck. These results illustrate BiRN's limitation in initial reward discovery, while its primary benefit lies in sustaining exploration by mitigating premature concentration on early rewarding behaviors.
We thank the reviewer for this constructive suggestion again. In response, we have added a corollary following Proposition 1 to provide further theoretical insights into BiRN's directional perturbation mechanism, and expanded the discussion of its applicability and limitations.
Response to the weaknesses
We sincerely thank the reviewer for the detailed and constructive feedback. We address the specific questions below:
BiRN is particularly relevant to sparse-reward settings because the scarcity of rewarding experiences can cause early accessible reward regions to dominate policy updates before alternative paths are sufficiently explored. By attenuating above-average targets and strengthening below-average targets among positive-TD-error transitions, BiRN regulates the relative reinforcement of these transitions, helping prevent early rewarding regions from dominating learning.
We agree that BiRN does not provide reward-free exploration. When all observed rewards and value estimates are zero, the TD errors are zero, and the perturbation rule is inactive. Although nonzero initial value estimates may produce nonzero TD errors, these signals may primarily reflect initialization or estimation errors rather than meaningful exploration guidance. Therefore, BiRN is not designed to discover the first reward in strictly zero-reward settings. Instead, it targets the subsequent exploration process, where initial rewarding experiences may cause the policy to concentrate prematurely around suboptimal regions.
The key advantage of BiRN over SrRN lies in its directional control. SrRN introduces random perturbations without controlling whether they strengthen or weaken the current learning signals. In contrast, BiRN uses relative target values to attenuate above-average targets while strengthening below-average targets among positively reinforced transitions. This helps reduce the excessive reinforcement of currently dominant regions while preserving incentives for alternative transitions.
Our grid exploration experiment provides empirical evidence for this mechanism. Although SrRN broadens state visitation, it remains concentrated around the suboptimal region with a return of 1. In contrast, BiRN substantially reduces repeated visits to this region (62 vs. 193 visits during the first 1k episodes) and guides exploration toward the optimal region with a return of 5.
Regarding RND, it provides intrinsic rewards independently of environmental rewards and can therefore facilitate exploration before the first extrinsic reward is discovered. BiRN serves a different purpose by regulating existing reward-driven learning signals. We view these approaches as complementary rather than competing methods. As also demonstrated by our experiments combining BiRN with RE3 and VCSE, BiRN can be integrated with intrinsic-reward-based exploration methods.