acceptodds
Under review as a conference paper at ICLR 2027

Uncertain Rewards Bring More Possibilities: Bidirectional Reward Noise for Exploration in Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) relies on reward-driven policy updates, yet sufficient exploration remains a critical challenge for discovering high-quality policies. Due to the tendency to pursue high rewards, agents are prone to becoming trapped around rewarding states encountered early in training, hindering exploration of potentially more valuable regions and leading to premature convergence. Existing exploration methods commonly introduce additional stochasticity or intrinsic incentives to encourage broader exploration. However, stochasticity alone lacks explicit directional control, while intrinsic rewards often require additional modeling components. In this paper, we propose Bidirectional Reward Noise (BiRN), a lightweight exploration mechanism that directly perturbs training rewards. By injecting bidirectional noise, BiRN achieves two complementary effects: negative noise mitigates excessive attraction toward dominant rewarding transitions, while positive noise strengthens incentives for transitions with positive learning potential. Extensive experiments show that BiRN consistently improves final performance across diverse RL benchmarks with minimal additional overhead, with particularly strong gains in challenging sparse-reward settings.

Primary area:
reinforcement learning
Keywords:
Deep Reinforcement Learning, Exploration, Reward Noise

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BiwjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a lightweight and easy-to-implement exploration approach that can be used with both model-free and model-based RL methods. The proposed approach injects perturbations into the reward function based on the TD error and the estimated state value relative to the batch of states collected to update the value model. The sign of the perturbation is determined to direct the agent toward underexplored regions that may contain promising states but currently have low target values. The authors provide extensive experiments in both single-agent and multi-agent environments and demonstrate improved performance compared to the baselines.

Critical strengths

The paper proposes a method to address the well-known challenge of exploration in RL. The authors provide extensive experiments across multiple domains and compare their method with recent baselines proposed to address the same challenge.

Critical weaknesses

My main concern is the motivation behind BiRN's claimed superiority over SrRN in sparse-reward environments. The authors argue that BiRN improves exploration by determining the direction of reward perturbations based on TD errors and the average bootstrap target of the batch. However, it is unclear why this mechanism should be particularly effective in sparse-reward settings. In environments where rewards are only given upon reaching the goal, all observed rewards may remain zero until the first successful trajectory is found. With zero-initialized value estimates, this would lead to zero TD errors, meaning that BiRN would not inject any reward noise. Even with nonzero initial value estimates, the TD errors and target differences may mainly reflect random initialization or estimation errors rather than useful information about promising unexplored states.

Therefore, it is unclear why using these values to determine the direction of reward perturbations should lead to better exploration than the direction-blind noise used in SrRN or even the RND method. The authors should provide a clearer theoretical explanation or empirical evidence supporting this mechanism, particularly by investigating how BiRN encourages exploration before the agent discovers its first reward in genuinely sparse-reward environments.

Questions for authors

1- Figure 1 lacks sufficient explanation. It is unclear which environment or experimental setting the figure represents, making its interpretation difficult. The authors should provide a more detailed explanation of the figure and clarify how it illustrates the proposed method.

2- The illustrative gridworld environment appears to be specifically designed to favor the proposed exploration method. Moreover, the additional results reported in the appendix show that count-based exploration achieves greater state coverage than the proposed method during early training and reaches nearly identical returns after 50K training episodes. These findings raise questions about the claimed exploration advantages of the proposed method.

3- The evaluation on Atari games is limited to only four environments. It would be valuable to evaluate the proposed method on more challenging sparse-reward exploration benchmarks, such as Montezuma's Revenge and Pitfall, to better assess its effectiveness in environments requiring substantial exploration.

4 comments
AuthorsOpenReview ↗

Response to the weaknesses

We sincerely thank the reviewer for the detailed and constructive feedback. We address the specific questions below:

However, it is unclear why this mechanism should be particularly effective in sparse-reward settings.

BiRN is particularly relevant to sparse-reward settings because the scarcity of rewarding experiences can cause early accessible reward regions to dominate policy updates before alternative paths are sufficiently explored. By attenuating above-average targets and strengthening below-average targets among positive-TD-error transitions, BiRN regulates the relative reinforcement of these transitions, helping prevent early rewarding regions from dominating learning.

We agree that BiRN does not provide reward-free exploration. When all observed rewards and value estimates are zero, the TD errors are zero, and the perturbation rule is inactive. Although nonzero initial value estimates may produce nonzero TD errors, these signals may primarily reflect initialization or estimation errors rather than meaningful exploration guidance. Therefore, BiRN is not designed to discover the first reward in strictly zero-reward settings. Instead, it targets the subsequent exploration process, where initial rewarding experiences may cause the policy to concentrate prematurely around suboptimal regions.

It is unclear why using these values to determine the direction of reward perturbations should lead to better exploration than the direction-blind noise used in SrRN or even the RND method. The authors should provide a clearer theoretical explanation or empirical evidence supporting this mechanism, particularly by investigating how BiRN encourages exploration before the agent discovers its first reward in genuinely sparse-reward environments.

The key advantage of BiRN over SrRN lies in its directional control. SrRN introduces random perturbations without controlling whether they strengthen or weaken the current learning signals. In contrast, BiRN uses relative target values to attenuate above-average targets while strengthening below-average targets among positively reinforced transitions. This helps reduce the excessive reinforcement of currently dominant regions while preserving incentives for alternative transitions.

Our grid exploration experiment provides empirical evidence for this mechanism. Although SrRN broadens state visitation, it remains concentrated around the suboptimal region with a return of 1. In contrast, BiRN substantially reduces repeated visits to this region (62 vs. 193 visits during the first 1k episodes) and guides exploration toward the optimal region with a return of 5.

Regarding RND, it provides intrinsic rewards independently of environmental rewards and can therefore facilitate exploration before the first extrinsic reward is discovered. BiRN serves a different purpose by regulating existing reward-driven learning signals. We view these approaches as complementary rather than competing methods. As also demonstrated by our experiments combining BiRN with RE3 and VCSE, BiRN can be integrated with intrinsic-reward-based exploration methods.

AuthorsOpenReview ↗

Response to the questions

Q1: Figure 1 lacks sufficient explanation. It is unclear which environment or experimental setting the figure represents, making its interpretation difficult. The authors should provide a more detailed explanation of the figure and clarify how it illustrates the proposed method.

Thank you for pointing this out. Figure 1 is intended as a conceptual illustration rather than a visualization of any specific environment or experimental setting. The figure illustrates the two complementary effects of BiRN: negative perturbations reduce the height of relatively high-reward regions that may attract premature exploitation, while positive perturbations strengthen alternative regions with relatively lower targets, provided that their TD errors are positive.

We agree that the original figure caption does not sufficiently explain these elements. We have revised the accompanying description to clarify the conceptual nature of the figure and the role of each directional perturbation.

Q2: The illustrative gridworld environment appears to be specifically designed to favor the proposed exploration method. Moreover, the additional results reported in the appendix show that count-based exploration achieves greater state coverage than the proposed method during early training and reaches nearly identical returns after 50K training episodes. These findings raise questions about the claimed exploration advantages of the proposed method.

The gridworld is designed as a controlled setting to examine the premature-reward-trap scenario targeted by BiRN, where an easily accessible suboptimal reward competes with a more distant, higher reward. This setting makes it possible to directly examine whether the proposed directional perturbation helps overcome premature exploitation.

BiRN's exploration advantage lies in reducing premature exploitation of suboptimal rewards, rather than maximizing overall state coverage. Although PPO-Count achieves slightly higher state coverage during early training (97.07% vs. 94.67%) and a similar final return, its broader coverage fails to translate into faster progress toward the higher-reward solution: it visits S2S_2 more than twice as often as BiRN (142 vs. 62 visits) and achieves lower returns at intermediate training stages. Thus, BiRN reduces repeated visits to suboptimal reward regions and accelerates progress toward higher-reward solutions.

Importantly, BiRN's applicability is not limited to this illustrative gridworld. It provides a lightweight and readily integrable exploration mechanism by reusing TD errors and bootstrap targets already available in Bellman-style RL algorithms, without requiring additional trainable components or modifications to network architectures. Its evaluation across discrete and continuous control, sparse-reward tasks, and multi-agent environments further demonstrates its applicability to diverse RL settings.

Q3: The evaluation on Atari games is limited to only four environments. It would be valuable to evaluate the proposed method on more challenging sparse-reward exploration benchmarks, such as Montezuma's Revenge and Pitfall, to better assess its effectiveness in environments requiring substantial exploration.

We appreciate the suggestion. Our evaluation aims to examine BiRN across diverse RL environments, including Atari games, continuous-control tasks, sparse-reward navigation, and multi-agent environments, rather than focusing exclusively on expanding the number of Atari games. Nevertheless, following the reviewer's suggestion, we extended our Atari evaluation by adding Pitfall and Montezuma's Revenge. Due to the limited time, each run was trained for 40M environment steps.

Pitfall. PPO+BiRN achieved the highest mean final game score of −0.342, compared with −0.726 for PPO. However, PPO, PPO+RND, and PPO+BiRN did not obtain positive extrinsic rewards within this training budget. Thus, BiRN's higher score primarily reflects fewer penalties rather than successful discovery of positive-reward trajectories.

MethodFinal game score ↑
PPO−0.726 ± 0.315
PPO + NoisyNet−0.607 ± 1.144
PPO + RLE−0.948 ± 0.286
PPO + RND−3.852 ± 0.418
PPO + BiRN−0.342 ± 0.081

Montezuma's Revenge. Under the same 40M-step budget, PPO, PPO+RND, and PPO+BiRN did not exhibit sustained reward growth or reliably discover rewarding states. Although occasional nonzero scores were observed, none of the methods achieved consistent progress.

These additional experiments extend our evaluation to two challenging sparse-reward Atari games. While BiRN achieves a higher final score on Pitfall and shows occasional positive-reward discovery on Montezuma's Revenge, reliably discovering and accumulating rewards remains challenging under the 40M-step budget. This is consistent with BiRN's role in regulating existing reward-driven learning signals, rather than providing reward-free exploration.

Reviewer_BiwjOpenReview ↗

Response to Authors’ Comments

Thank you for providing explanations in response to my comments. I was expecting more theoretical justification for why your approach would be beneficial in sparse-reward environments, where the agent might not encounter a reward until much later in training, especially when the goal is difficult to reach. This is also consistent with what the authors showed empirically in the Montezuma's Revenge environment.

Overall, I maintain my positive score, but I was not satisfied with your justification regarding RND and the comparison between BiRN and SrRN. Since your method aims to improve exploration, it should either outperform the other baselines or at least demonstrate reasonable performance in long-horizon, sparse-reward environments.

Thank you for encouraging us to provide a deeper theoretical analysis of BiRN under sparse rewards.

Our key observation is that sparse rewards make successful feedback both rare and concentrated, which can cause a small number of behaviors to dominate the learning signal. Consider a state ss with two actions aia_i and aja_j, whose subsequent trajectories reach the same terminal goal with probabilities pip_i and pjp_j. If mm independent trajectories are sampled after each action, let qk=1−(1−pk)mq_k=1-(1-p_k)^m be the probability of observing at least one success for action kk. Conditional on observing any success, the probability that only one action produces a success is qi(1−qj)+qj(1−qi)qi+qj−qiqj\frac{q_i(1-q_j)+q_j(1-q_i)}{q_i+q_j-q_iq_j}. This approaches 11 when mpi,mpj≪1mp_i,mp_j\ll1. Thus, even when two actions are comparably promising, a success, when observed under sparse feedback, is likely to come from only one action. The probability of observing any success, qi+qj−qiqjq_i+q_j-q_iq_j, also decreases as the goal becomes harder to reach.

Proposition 1 and BiRN's perturbation rule show how directional perturbation reduces the expected target gap when sparse reward feedback is concentrated on a few behaviors. Consider transitions i,j∈Bi,j\in\mathcal B from this decision state, with i∈B+i\in\mathcal B^+ and yi>yˉB>yjy_i>\bar y_{\mathcal B}>y_j. Under the perturbation analyzed in Proposition 1, E[y~i−y~j∣B]=yi−yj−λ2/π(1+1{δj>0})\mathbb E[\tilde y_i-\tilde y_j\mid\mathcal B]=y_i-y_j-\lambda\sqrt{2/\pi}\bigl(1+\mathbf 1\{\delta_j>0\}\bigr), whereas SrRN leaves the expected gap unchanged. Thus, once a rare success is reflected in a positive TD signal for ii while δj=0\delta_j=0, BiRN reduces the relative reinforcement of ii. If value information subsequently propagates to a transition following aja_j, giving it δj>0\delta_j>0 despite zero immediate reward, BiRN also strengthens this alternative transition. The expected gap reduction therefore increases from λ2/π\lambda\sqrt{2/\pi} to 2λ2/π2\lambda\sqrt{2/\pi}. Under a local two-action softmax policy-gradient update using TD errors as fixed advantages, this smaller gap reduces the update favoring aia_i and increases the probability of trying aja_j to first order.

This analysis also clarifies the distinct roles and limitations of BiRN relative to RND and SrRN. RND provides reward-independent exploration signals and can therefore encourage exploration before any extrinsic reward is observed. BiRN, in contrast, relies on reward-derived TD signals and remains inactive when both observed rewards and value estimates are zero. However, once sparse reward feedback begins to shape the learning targets, BiRN counteracts the disproportionate influence of early rewarding transitions. Unlike SrRN, whose zero-mean perturbations leave the expected target gap unchanged, BiRN directionally adjusts the relative learning signals and can further strengthen alternative transitions as they acquire positive TD errors. In our 40M-step Montezuma's Revenge runs, PPO and PPO+BiRN occasionally obtained positive scores, but neither showed sustained reward growth, suggesting that reliably discovering and reinforcing rewarding trajectories remained a bottleneck. These results illustrate BiRN's limitation in initial reward discovery, while its primary benefit lies in sustaining exploration by mitigating premature concentration on early rewarding behaviors.

We thank the reviewer for this constructive suggestion again. In response, we have added a corollary following Proposition 1 to provide further theoretical insights into BiRN's directional perturbation mechanism, and expanded the discussion of its applicability and limitations.

Reviewer_GfZkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Bidirectional Reward Noise (BiRN), a lightweight exploration method that perturbs training rewards using TD errors and batch relative target values. It reduces the reinforcement of high target transitions while boosting potentially promising alternatives, without adding trainable components. A conditional analysis describes the resulting shift in learning targets, and experiments on gridworld, Atari, IsaacGym, MiniGrid, and cooperative multi agent tasks report improved exploration or returns, particularly in sparse reward settings.

Critical strengths

1.BiRN uses TD errors and bootstrap targets already available during training, without adding a trainable exploration model or changing the policy architecture. Appendix Table 4 reports approximately 1.13% average additional runtime on the Atari tasks. This makes the method practical as an add-on to existing algorithms. 2.The evaluation covers gridworld, Atari, IsaacGym, MiniGrid, and cooperative multi-agent tasks. The gridworld experiment is particularly informative: during the first 1,000 episodes, BiRN visits the nearby suboptimal reward state 62 times, compared with 484 visits for PPO, while later reaching the higher-reward state. The MiniGrid results also suggest that BiRN can complement A2C, RE3, and VCSE rather than depending on one particular backbone.

Critical weaknesses

1.Proposition 1 shows how the perturbed target moves relative to the batch mean when the batch and value estimates are fixed. It does not establish that this movement increases useful state visitation or improves policy return. In particular, a large TD error need not indicate underexploration, and an above-average target need not represent a suboptimal reward attractor. Moreover, the proposition uses unbounded half-normal noise, while the implemented method clips the noise to [-1,1]; its exact expectation and variance therefore do not describe the implemented perturbation. 2.The initial noise scale is chosen separately for different tasks, but the paper does not provide a complete comparison under matched tuning budgets. Although it includes positive- and negative-noise ablations on Gravitar, it lacks systematic comparisons against a deterministic bidirectional reward shift and perturbations matched for magnitude and schedule. Some MiniGrid results have wide reported intervals, while the Atari, IsaacGym, and multi-agent results are presented mainly as curves without per-seed final scores. These gaps make it difficult to determine how consistently BiRN helps and whether stochastic noise itself is necessary.

Questions for authors

1.Is stochastic reward noise necessary for the reported gains? How would BiRN compare with a deterministic bidirectional reward adjustment that has the same expected magnitude and decay schedule? This comparison would clarify whether the benefit comes from noise or from the directional shift in learning targets. 2.How reliable are the signals used to choose the perturbation direction? Why should a positive TD error with a below-average target indicate a promising underexplored transition, while an above-average target indicates an early reward attractor? In what situations might these rules instead boost poor actions or suppress optimal ones? 3.Do the performance gains on the main benchmarks result from improved exploration? The gridworld and one MiniGrid task include visitation evidence, but the Atari, IsaacGym, and multi-agent results primarily report returns. Can the authors provide behavioral evidence in representative tasks showing that BiRN discovers useful states or strategies that the baselines miss?

4 comments
AuthorsOpenReview ↗

Response to the weaknesses

We sincerely thank the reviewer for the detailed and constructive feedback. We address the specific comments below:

Moreover, the proposition uses unbounded half-normal noise, while the implemented method clips the noise to [-1,1]; its exact expectation and variance therefore do not describe the implemented perturbation.

We thank the reviewer for pointing out this aspect of the theoretical analysis. Proposition 1 characterizes the directional effect under the unclipped half-normal perturbation. We agree that extending the analysis to the clipped perturbation used in implementation provides a more complete characterization.

Specifically, for each selected transition i∈B+i\in\mathcal B^+, let Ci=min⁡(ϵi+,1)C_i=\min(\epsilon_i^+,1), where ϵi+∼N+(0,1)\epsilon_i^+\sim\mathcal N^+(0,1). Its mean and variance are denoted by μC=E[Ci]=2[ϕ(0)−ϕ(1)+1−Φ(1)]\mu_C=\mathbb E[C_i]=2[\phi(0)-\phi(1)+1-\Phi(1)] and vC=Var⁡(Ci)=1−2ϕ(1)−μC2v_C=\operatorname{Var}(C_i)=1-2\phi(1)-\mu_C^2, respectively. Replacing ϵi+\epsilon_i^+ with CiC_i in the derivation of Proposition 1 yields E[(y~iclip−yˉB)sgn⁡(yi−yˉB)∣B]=∣yi−yˉB∣−λμC\mathbb E[(\tilde y_i^{\mathrm{clip}}-\bar y_{\mathcal B})\operatorname{sgn}(y_i-\bar y_{\mathcal B})\mid\mathcal B]=|y_i-\bar y_{\mathcal B}|-\lambda\mu_C and Var⁡(y~iclip∣B)=λ2vC\operatorname{Var}(\tilde y_i^{\mathrm{clip}}\mid\mathcal B)=\lambda^2v_C. Thus, clipping changes the magnitude statistics while preserving the directional effect of BiRN. We have incorporated this additional analysis into the revised appendix.

The initial noise scale is chosen separately for different tasks, but the paper does not provide a complete comparison under matched tuning budgets.

The initial noise scale λ\lambda is a single scalar that controls the perturbation magnitude and is selected through a coarse search over a common candidate set {0.001, 0.01, 0.1, 0.5, 1} to account for differences in reward scales across environments. Importantly, the same λ=0.5\lambda=0.5 is used across all MiniGrid tasks and multi-agent tasks. Furthermore, sensitivity analyses show that BiRN consistently outperforms the corresponding baselines across all tested noise scales on both Gravitar and cooperative navigation.

We emphasize that this task-dependent calibration involves only one scalar parameter, while BiRN introduces no additional trainable components, auxiliary estimators, or network modifications. This represents a limited tuning requirement for a lightweight exploration mechanism that can be readily integrated into different RL algorithms.

These gaps make it difficult to determine how consistently BiRN helps and whether stochastic noise itself is necessary.

To provide a clearer assessment of BiRN's performance across tasks, we additionally report the final scores of the runs shown in Figs. 4 and 5 as mean ± standard deviation across seeds, together with the percentage improvement over PPO.

Final game score on Atari

EnvironmentPPORNDNoisyNetRLESrRNBiRNBiRN vs. PPO
Asterix5586.50 ± 738.594346.57 ± 509.586106.30 ± 741.666035.30 ± 1,158.336835.76 ± 576.246170.33 ± 409.46+10.45%
Amidar1104.01 ± 125.391006.28 ± 77.731051.75 ± 306.201062.17 ± 106.321284.69 ± 187.601434.91 ± 235.92+29.97%
Ms. Pac-Man3048.65 ± 553.821598.75 ± 206.823676.13 ± 289.752297.69 ± 270.163770.76 ± 688.463793.16 ± 177.19+24.42%
Gravitar1560.17 ± 355.71748.83 ± 114.161451.36 ± 784.811618.39 ± 549.342029.01 ± 154.852265.35 ± 138.63+45.20%

Final episode return on IsaacGym

EnvironmentPPORNDRLESrRNBiRNBiRN vs. PPO
Humanoid5410.13 ± 86.513708.27 ± 180.885428.98 ± 262.645557.60 ± 45.025475.79 ± 106.26+1.21%
Ant5660.25 ± 768.155229.19 ± 683.455557.65 ± 1102.696558.73 ± 319.946644.27 ± 294.73+17.38%
AnymalTerrain7.69 ± 1.229.07 ± 0.368.61 ± 0.958.70 ± 0.569.96 ± 0.48+29.48%

BiRN achieves higher mean returns than PPO across all seven tasks and obtains the highest mean among the compared methods in five of them. The improvements are observed across both discrete and continuous control tasks, although their magnitudes vary across environments. These additional statistics provide a clearer assessment of BiRN's final performance.

AuthorsOpenReview ↗

Response to the questions (Part 1)

Q1: Is stochastic reward noise necessary for the reported gains? How would BiRN compare with a deterministic bidirectional reward adjustment that has the same expected magnitude and decay schedule? This comparison would clarify whether the benefit comes from noise or from the directional shift in learning targets.

To examine whether stochastic perturbations provide benefits beyond the expected directional shift, we conducted additional experiments on Gravitar and Empty-25x25. Specifically, Fixed-shift retains the same transition selection, bidirectional sign rule, and decay schedule as BiRN, but replaces the sampled perturbation magnitude λtCi\lambda_t C_i with its expectation λtμC\lambda_t\mu_C. Thus, the two variants differ only in whether the perturbation magnitude is stochastic.

Empty-25x25 · Test return

Base algorithmNo perturbationFixed-shiftBiRN
A2C0.12 ± 0.010.10 ± 0.010.74 ± 0.42
RE30.14 ± 0.020.53 ± 0.500.76 ± 0.40
VCSE0.54 ± 0.510.76 ± 0.440.98 ± 0.01

Gravitar · Game score

Base algorithmNo perturbationFixed-shiftBiRN
PPO1560.17 ± 355.711760.42 ± 842.062265.35 ± 138.63

The results show that Fixed-shift improves over the unperturbed baselines with RE3, VCSE, and PPO, indicating that deterministic directional adjustment itself can be beneficial. However, BiRN achieves higher average returns in all four comparisons. Notably, Fixed-shift fails to improve over A2C on Empty-25x25, whereas BiRN achieves a substantially higher average return.

Although both methods preserve the same expected directional shift, stochastic perturbations introduce additional variation in value targets and the resulting updates. Through successive policy updates, these stochastic variations can influence the state-visitation distribution and create different exploration trajectories. These results suggest that stochastic perturbations provide additional benefits beyond the deterministic adjustment. We have included these additional results and the corresponding discussion in the appendix.

Q2: How reliable are the signals used to choose the perturbation direction? Why should a positive TD error with a below-average target indicate a promising underexplored transition, while an above-average target indicates an early reward attractor? In what situations might these rules instead boost poor actions or suppress optimal ones?

Regarding the reliability of the signals, BiRN determines the perturbation direction based on the relative positions of target values within the current batch, rather than their absolute magnitudes. Consequently, a common additive shift or positive rescaling of all target estimates does not change the direction assignment. Specifically, a positive TD error identifies transitions with upward value-correction signals, while comparing their targets against the batch mean determines whether these updates are strengthened or attenuated. This provides a relative criterion for regulating learning reinforcement without relying on an absolute target threshold.

Regarding the rationale behind the directional rule, a positive TD error restricts perturbation to transitions whose current value estimates are below their bootstrap targets. Within this subset, the batch mean provides a reference for regulating their relative target magnitudes. As characterized by Proposition 1 in the unclipped setting, BiRN shifts above-average targets downward and below-average targets upward in expectation, reducing their relative dominance. This is particularly relevant to premature reward attraction, where repeated reinforcement of easily accessible high-reward regions can cause learning to concentrate before alternative paths are sufficiently explored. By moderating these relatively high-target updates while strengthening other positive-TD-error transitions, BiRN helps preserve learning opportunities for alternative paths.

Regarding potential failure cases, the adjustment may be less beneficial when relative target values do not align with the long-term returns of the corresponding transitions. For example, a below-average target may correspond to a transition with limited long-term value, in which case positive perturbation may provide unnecessary reinforcement. Conversely, an above-average target may correspond to a genuinely valuable path, and negative perturbation may weaken its learning signal. Inaccurate value estimates may also contribute to such mismatches, particularly during early training. These situations may reduce the effectiveness of directional perturbation, depending on how frequently such mismatches occur. Meanwhile, BiRN's clipped perturbations and decaying noise scale help limit their potential impact by bounding the magnitude of each intervention and progressively reducing its influence during training.

AuthorsOpenReview ↗

Response to the questions (Part 2)

Q3: Do the performance gains on the main benchmarks result from improved exploration? The gridworld and one MiniGrid task include visitation evidence, but the Atari, IsaacGym, and multi-agent results primarily report returns. Can the authors provide behavioral evidence in representative tasks showing that BiRN discovers useful states or strategies that the baselines miss?

Yes. We provide additional behavioral evidence from representative Atari and multi-agent tasks, showing that BiRN expands state visitation and discovers effective strategies that the baselines fail to learn.

  • Gravitar. At the shared 5M-step checkpoint, BiRN visited 240 distinct discretized observations, compared with 215 for PPO, under the same budget of 10000 evaluation observations. This provides evidence that BiRN explores a broader range of states than the baseline.

  • 12-step matrix game. Action logs reveal different exploration and strategy discovery behaviors between BiRN and SrRN. At approximately 9k training steps, BiRN repeatedly had both agents choose action 0, whereas SrRN repeatedly had both agents choose action 1. By approximately 18k steps, BiRN had discovered the optimal coordination strategy of having both agents switch to action 1 on the final step, while SrRN retained its original strategy throughout the remaining training. This shows that BiRN discovered a coordination strategy that SrRN failed to explore.

  • Endangered Wildlife Rescue. Under the same 60k-episode training budget, BiRN rescued both animals in 97.5% of the final training episodes, compared with 49.0% for SrRN and 0% for MADDPG. This demonstrates that BiRN learned more effective coordination behavior, rather than consistently following the less effective strategy of rescuing only the less severely injured animal, which terminates the episode.

These results provide behavioral evidence of BiRN's exploration benefits, including broader state visitation and the discovery of more effective coordination strategies that are not observed with the baselines.

Thank you again for your constructive feedback. As the discussion period draws to a close, we would be grateful if you could reconsider your assessment in light of our responses and the revised manuscript. If any concerns remain, we would be happy to provide further clarification.

Reviewer_YV2KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BiRN, a reward perturbation scheme for exploration. For transitions with positive TD error, BiRN adds half-normal noise to the reward. The sign is chosen so that bootstrap targets above the batch mean are pushed down and those below it are pushed up, and the noise scale decays over training. A simpler variant, SrRN, adds zero-mean Gaussian noise to the transitions with the largest absolute TD error. Both are combined with PPO, A2C, intrinsic-reward methods and multi-agent learners, and evaluated on a grid example, Atari, IsaacGym, MiniGrid and several cooperative multi-agent tasks.

Critical strengths

The method is cheap and easy to add to existing algorithms, including intrinsic-reward methods. Some of the gains are large, most clearly on Gravitar, Empty-25x25 and the multi-agent matrix game. The grid example also shows nicely how the method keeps the agent from settling on a nearby small reward. If the mechanism holds up, this would be a useful trick.

Critical weaknesses
  • It is not clear to me that the "noise" is the active ingredient, and this is what my rating mostly depends on. As Prop. 1 shows, BiRN's expected effect is a deterministic shift of positive-TD targets toward the batch mean, with some noise on top. A version with the same sign rule and a fixed shift is never tried, so whether BiRN's gains come from the noise or from the shift is not tested. The comparison with SrRN does not settle it either, since the two differ in which transitions are perturbed, the sign rule and the tuned λ at once. The component ablation (Fig. 7a, Gravitar) also does not quite match the motivation. Much of the paper's story is about negative noise tempering early rewards, yet removing that part costs far less than removing the positive part.

  • The claims of consistent gains, especially in sparse-reward settings, go beyond what the curves show. Gravitar is the only sparse-reward Atari game. On IsaacGym the margins are modest and the bands overlap, despite the Fig. 5 caption saying the methods "outperform the other baselines". In MiniGrid the gains on MultiRoom and Fetch are small. The noise scale λ is also chosen per task, while the baselines run with fixed settings, and Fig. 7c suggests a good part of the Gravitar margin depends on that choice. I would either add a few harder sparse or distractor tasks with λ fixed per domain, or narrow the claims.

Questions for authors
  1. It would help to compare BiRN with the fixed-shift control described above on a couple of tasks, e.g. Gravitar and Empty-25x25.
  2. On Empty-25x25, the returns reported for the BiRN variants in Table 1 are higher than their success rates in Table 5. That should not be possible with MiniGrid's reward if both are computed on the same episodes. How were the two tables obtained?
5 comments
AuthorsOpenReview ↗

Response to the weaknesses

We sincerely thank the reviewer for the detailed and constructive feedback. We address the specific questions below:

W1: The component ablation (Fig. 7a, Gravitar) also does not quite match the motivation. Much of the paper's story is about negative noise tempering early rewards, yet removing that part costs far less than removing the positive part.

The return ordering in Fig. 7(a) is compatible with our motivation because the two ablations can fail at different stages of learning. An agent that repeatedly collects an easily accessible suboptimal reward can earn a higher return than one that discovers a better goal but rarely reaches it again. The negative branch is designed to weaken early commitment to the suboptimal reward and broaden exploration. The positive branch is designed to reinforce promising transitions once they are sampled, so that the agent can learn to repeat behaviors that lead to higher rewards.

To further illustrate the complementary roles of the two branches, we added a component ablation in our two-reward grid exploration environment. The agent starts closer to the suboptimal state S2S_2 (reward +1+1) than to the optimal state S1S_1 (reward +5+5), so the reward at S2S_2 can dominate early learning.

VariantVisits to suboptimal state S2S_2 @1kState coverage@1kFirst visit to optimal state S1S_1 (episode)Return@50k
Full BiRN62.394.67%416.74.955
w/o negative noise203.580.44%Unreached within 50k episodes0.995
w/o positive noise46.094.44%735.50.046

The two ablated variants have the same return ordering in the grid exploration environment as in Fig. 7(a). Without negative noise, the agent visited S2S_2 more often, covered less of the state space, and earned a final mean return of 0.995 by repeatedly collecting the suboptimal reward. It never reached S1S_1 within 50000 episodes. With negative noise alone, the agent explored more broadly and discovered S1S_1, but its final mean return remained low at 0.046. With both branches, BiRN achieved a final mean return of 4.955, close to the optimal payoff of 5.

These results illustrate why the two branches complement each other. When an easily accessible reward dominates early learning, moving beyond the resulting suboptimal strategy is a prerequisite for further improvement. Reinforcing promising behavior once it is sampled can then help turn an occasional discovery into sustained gains. We have added new ablation results in the appendix.

W2: The claims of consistent gains, especially in sparse-reward settings, go beyond what the curves show.

Regarding the performance claims, we would like to clarify that our claims of consistent gains refer to improvements over the corresponding base algorithms when BiRN is integrated. Our primary objective is providing a lightweight exploration mechanism that can be readily integrated into existing RL algorithms without architectural changes. Although Gravitar is the only sparse-reward game in our Atari suite, our evaluation also covers sparse-reward MiniGrid and multi-agent tasks. In MiniGrid, BiRN yields higher mean returns in all nine task–algorithm comparisons, although the gains vary in magnitude. On IsaacGym, the mean differences are modest, and we have therefore revised the Fig. 5 caption to avoid implying superiority over all baselines.

Regarding noise-scale selection, BiRN introduces only one additional scalar parameter, λ\lambda, which controls the perturbation magnitude relative to the reward scale. We use a common candidate set {0.001, 0.01, 0.1, 0.5, 1} for coarse-grained selection on Atari and IsaacGym, while keeping the other base algorithm configurations unchanged in each paired comparison. Moreover, the same λ=0.5\lambda=0.5 is used across all MiniGrid and multi-agent tasks. Although the magnitude of the gains varies with λ\lambda, Fig. 7(c) shows that BiRN achieves higher final mean scores than PPO at all three tested scales on Gravitar. These results indicate that its gains are not confined to a single narrowly selected noise scale.

AuthorsOpenReview ↗

Response to the questions

Q1: It would help to compare BiRN with the fixed-shift control described above on a couple of tasks, e.g. Gravitar and Empty-25x25.

Thank you for this thoughtful suggestion. Distinguishing the contribution of stochastic perturbations from that of the expected directional shift is essential to understanding the effectiveness of BiRN. To address this question, we conducted additional experiments on Gravitar and Empty-25x25, comparing BiRN with a deterministic fixed-shift control.

Specifically, Fixed-shift retains the same transition selection, bidirectional sign rule, and decay schedule as BiRN, but replaces the sampled clipped half-normal perturbation magnitude λtCi\lambda_t C_i with its expectation λtE[Ci]\lambda_t\mathbb{E}[C_i], where Ci=min⁡(ϵi+,1)C_i=\min(\epsilon_i^+,1) and ϵi+∼N+(0,1)\epsilon_i^+\sim\mathcal{N}^+(0,1). Thus, the two variants differ only in whether the perturbation magnitude is stochastic or deterministic.

Empty-25x25 · Test return

Base algorithmNo perturbationFixed-shiftBiRN
A2C0.12 ± 0.010.10 ± 0.010.74 ± 0.42
RE30.14 ± 0.020.53 ± 0.500.76 ± 0.40
VCSE0.54 ± 0.510.76 ± 0.440.98 ± 0.01

Gravitar · Game score

Base algorithmNo perturbationFixed-shiftBiRN
PPO1560.17 ± 355.711760.42 ± 842.062265.35 ± 138.63

BiRN achieves the highest average return in all four comparisons, outperforming the deterministic Fixed-shift variant across both environments and all three MiniGrid base algorithms. Notably, Fixed-shift also improves over the baseline with RE3, VCSE, and PPO, suggesting that the expected directional shift itself contributes to performance improvement. However, BiRN further improves the average return in all four settings. This difference is particularly evident with A2C on Empty-25x25, where Fixed-shift fails to improve over the baseline, and on Gravitar, where BiRN achieves a higher average score than Fixed-shift.

These results suggest that stochastic perturbations provide additional benefits beyond the deterministic adjustment. We have added these additional results in the revised appendix.

Q2: On Empty-25x25, the returns reported for the BiRN variants in Table 1 are higher than their success rates in Table 5. That should not be possible with MiniGrid's reward if both are computed on the same episodes. How were the two tables obtained?

The test returns in Table 1 and the success rates in Table 5 come from separate evaluations of policies from different training runs. Table 1 reports test returns from the original benchmark runs. For Table 5, we reran the Empty-25x25 experiments to record state visitation during training and evaluated the final checkpoints from those reruns to obtain the success rates in Table 5. We have revised Table 5 to clarify the source of the reported results.

Thank you again for your constructive feedback. As the discussion period draws to a close, we would like to follow up on the fixed-shift comparison you suggested. With the expected perturbation matched to BiRN, BiRN achieved higher mean performance in all four comparisons across Gravitar and Empty-25x25. We have added these results to the revised manuscript and would be grateful if you could consider them when revisiting your assessment. If any concerns remain, we would be happy to clarify them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.