acceptodds
Under review as a conference paper at ICLR 2027

Different Bonuses, Similar Outcomes: Understanding Auxiliary Reward Non-Separation in PPO

Abstract

Auxiliary-reward methods are usually distinguished by the quantity that generates their bonus, yet the bonus itself is not the object optimized by a policy-gradient method. We study a general learning question: how can mathematically different auxiliary objectives produce nearly indistinguishable behaviour after integration into the same policy-optimization pipeline? Six PPO-based systems are compared in a controlled dynamic-survival task: PPO alone, intrinsic curiosity (ICM), random-network distillation (RND), hashed pseudocount exploration, advisory planning, and ICM with advisory planning. Across 40 matched seeds, 240 policies are trained in 120,000 episodes and evaluated in 336,000 held-out episodes. Raw intrinsic signals remain clearly distinguishable; their mean per-step magnitudes span more than three orders of magnitude, while stochastic test survival occupies only a 32.63% to 33.25% range. The hybrid is numerically first, but its paired effects are only to percentage points; every Holm-adjusted survival test has , and every confidence interval excludes the prespecified five-point benefit. The ICM-by-planner interaction is also unsupported. We formalize an effective-update view in which generalized advantage estimation, episodewise standardization, PPO clipping, and mapping through policy score vectors form a many-to-one map from raw auxiliary rewards to actor updates. The experiment does not establish equal gradients or general algorithmic equivalence. It does establish a controlled case in which semantic and numerical differences among auxiliary signals fail to yield practically distinguishable downstream outcomes.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, proximal policy optimization, intrinsic motivation, exploration, auxiliary rewards, reward shaping, policy gradients, safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_65uqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper performs a controlled experiment to examine whether different bonus rewards can produce similar results when used with the same reinforcement learning algorithm. It compares six versions of PPO in a survival game, with bonuses encouraging different behaviours. The paper shows that although encouraged to act different - the outcomes of all the versions is roughly the same 33% survival rate.

Critical strengths

The controlled comparison across 40 matched seeds is a substantial. Holding the PPO architecture and training configuration fixed, together with evaluation on held-out maps, makes the comparison more reliable.

Critical weaknesses
  1. The comparison is restricted to one environment, one training budget, and fixed bonus coefficients. The authors also acknowledge that the curiosity bonuses are tiny relative to the environmental reward. Consequently, the similar performance may depend on this particular configuration.

  2. The paper describes how PPO could suppress differences between auxiliary rewards, but does not measure the advantages, clipping activity, or gradients needed to investigate those mechanisms. Similar final survival rates cannot distinguish similar updates from different learning processes that reach similar outcomes. Although the authors acknowledge this limitation, it leaves the central explanatory contribution unsupported by direct empirical evidence.

Questions for authors
  1. Can the authors demonstrate a similar effect on more environments?

  2. Can the authors suggest a concrete change to the reward design or PPO training process that could make the auxiliary rewards more effective in this setting? Which proposed explanation for the similar outcomes would this change address?

  3. Could you provide a small coefficient or bonus-normalization ablation? For the recorded bonuses which are very small relative to the task-reward terms and are further reduced by their coefficients.

1 comment
AuthorsOpenReview ↗

Reward Calibration and Additional Experiments - Response to Reviewer 65uq

We thank the reviewer for requesting reward-scale calibration, independent-environment evaluation, and a more concrete investigation of auxiliary-reward effectiveness. We have revised the manuscript and conducted additional experiments addressing these points.

1. Reward-scale intervention (Section 5.1; Appendix G, Tables 12–14). We calibrated three RND multipliers on separate development seeds and evaluated them over ten paired training seeds. Increasing RND strength produced measurable differences in normalized advantages and actor gradients (gradient-influence contrasts: Holm-adjusted p ≈ 0.0296). However, the observed GridWorld survival changes were −0.48, −2.00, and −1.47 percentage points relative to original RND, with no established improvement.

This provides a concrete intervention and an important qualification: increasing an initially weak auxiliary reward changes the learning signal, but stronger influence does not necessarily improve task performance. Measuring achieved reward strength and actor-gradient influence can therefore help distinguish ineffective scaling from a lack of demonstrated downstream benefit.

2. Independent-environment evaluation (Section 6.1; Appendix J, Table 24). In MiniGrid-FourRooms, original-scale RND achieved 53.6% mean success versus 40.8% for PPO across five paired development seeds (+12.8 percentage points). This observed gain did not satisfy the prespecified exact-test superiority rule (Holm-adjusted p = 0.25), and we do not claim general superiority.

3. PPO mechanism correction (Section 5.3; Appendix F.5). We verified that active clipping was absent across 30,000 measured original-style updates and removed clipping as an explanation for the original results.

The revised paper now distinguishes reward magnitude, optimization-level influence, learned-policy differences, and task outcomes, while acknowledging unresolved generalization.

We would appreciate the reviewer's assessment of whether these additional experiments and corrected interpretations address the principal concerns.

Reviewer_MseuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether mathematically differing auxiliary reward instantiations yield differing outcomes when incorporated into identical PPO pipelines. The authors compose six PPO-based algorithms with varying types of auxiliary rewards including ICM, RND, pseudo count-based exploration, among others for testing. Testing is done in a grid world survival environment designed by the authors across 40 seeds. In their experiments, the authors find minimal difference in survival rate in this environment across the auxiliary reward specifications despite their dissimilarities in formulation. Alongside their empirical results, the authors also describe possible mechanisms within PPO's algorithm that could be the source of minimal differences between auxiliary reward formulations when measured by survival outcomes.

Critical strengths

The experimental comparisons given in this paper are relatively extensive in large part due to the fact the authors focus on a specific implementation of grid world, allowing them to run many seeds and experimental variations without much compute. The relatively high number of seeds adds trustworthiness to the results that a reader may not have given fewer seeds.

The authors provide an update formulation in Section 3 which could prove useful in understanding why various flavors of auxiliary rewards might produce relatively similar policy updates when passed through PPO's machinery. The authors identify multiple optimization knobs that could reduce the differences induced by different auxiliary reward choices.

Critical weaknesses

The paper is very poorly motivated and has very unclear broader significance. I appreciate the authors conducting a relatively thorough study with controlled experiments and in a relatively isolated way using PPO, but without the motivation, why should a reader pick up this paper and read it? The central question as identified in the introduction is not self-contained, poorly motivated, and it is never made clear why a practitioner should care about the contributions of this work. This concern extends to the experimental results sections as well. Why is this particular instantiation of a grid world used for experimentation here? There is no motivation as to why this particular version of grid world is especially informative and what this choice of grid world adds to your analysis. Due to these issues, the broader implications of this work are unclear to me.

The paper demonstrates that regardless of evaluated auxiliary reward, the outcomes have similar downstream survival rates in their experimental environment, but as far as I can tell the authors don't establish why this is. The formulation in Section 3 provides some possible mechanisms, but nothing substantial to back a causal relationship. This same complaint is prevalent through the paper. Often times the authors provide many factual statements, many numbers, and many textual descriptions of results, but minimal analysis or further development of the ideas. There is a lot of writing and reported results in this paper, but not much to learn. Many of the numbers and sentences should have been put in the appendix for the curious reader rather than reported in the main text (e.g. most of section 4.2 could have simply been left to Appendix D). With that extra space, the authors could have delved much deeper into analysis and even more empirical results.

Questions for authors

Could the authors clarify the motivation for why studying the behavior of these auxiliary mechanism is useful and why the focus was on PPO instead of other RL algorithms? For a practitioner, what are the practical takeaways from the empirical results of this work? How should they adjust their instantiations of PPO?

Why this version of grid world? What about it is especially informative compared to simpler instantiations? What would we lose out on by removing various components in evaluating the various auxiliary rewards?

1 comment
AuthorsOpenReview ↗

Motivation and Scientific Contribution - Response to Reviewer Mseu

We thank the reviewer for highlighting weaknesses in the original motivation, practical relevance, and scientific interpretation. We agree that reporting similar final survival scores without investigating the underlying learning processes provides limited insight.

Our revised contribution is a concrete diagnostic distinction: when an auxiliary reward fails to improve performance, did it meaningfully influence optimization, and did it change the learned policy?

The revised experiments identify two situations that final scores alone cannot distinguish.

First, the original ICM and RND configurations produced very small weighted reward contributions at measured checkpoints. Second, increasing RND strength in a ten-seed intervention measurably changed normalized advantages and actor gradients, while independently trained policies also differed on shared sampled observations. Despite these differences, no reliable GridWorld survival improvement was established (Sections 5.1–5.2; Appendix G).

Practical implication (Section 7.1). We now propose evaluating unsuccessful auxiliary rewards through three stages: (1) measure achieved weighted reward strength, (2) measure changes in normalized advantages and actor gradients using controlled comparisons, and (3) examine learned-policy differences alongside held-out performance and training-seed-level uncertainty.

This helps distinguish insufficient reward influence from meaningful learning changes that do not demonstrate task benefit. Increasing reward magnitude may be diagnostically informative without being an effective performance intervention.

We also expanded the GridWorld motivation (Section 2.1), explaining how stochastic hazards, terrain costs, state visitation, and planner recommendations provide opportunities for different auxiliary mechanisms to affect learning. We acknowledge that component-level ablations are not established.

Finally, we reorganized the paper around this diagnostic interpretation, corrected the original PPO clipping explanation, and added independent MiniGrid investigations while explicitly limiting generalization claims.

We would appreciate the reviewer's assessment of whether the revised motivation, learning measurements, and practical interpretation address the concerns about informativeness and scientific contribution.

Reviewer_ZB1VOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows that different auxiliary-reward methods can lead to the same performance during RL training. They compare standard PPO with five variants: curiosity, random-network distillation, pseudo-count exploration, planner advice, or a combination of curiosity and a planner, on a custom Gridworld survival task and observe that they all achieve similar survival rates of around 33%. The authors then hypothesize which steps in the PPO algorithm might lead to the same outcome under different rewards, arguing that aspects of the PPO update, such as advantage estimation, normalization, and clipping, can reduce differences between learning signals.

Critical strengths

The chosen auxiliary reward methods cover a diverse range of ways to construct auxiliary rewards.

Critical weaknesses
  • The experiments use one custom gridworld family, one PPO configuration, and a fixed training budget, without varying auxiliary-reward coefficients or calibrating bonus scales. First, this confounds what we observe, since it is unclear whether PPO is tuned for this environment. Moreover, without varying bonus strengths or training budgets, it is unclear whether the similar results reflect PPO’s treatment of these rewards or settings that are just not optimal for the improvement,especaily given that ICM and RND bonuses are small relative to environmental rewards.
  • The authors discuss how advantage normalization and PPO clipping could reduce differences between rewards, but do not measure their effects on policy updates or test these explanations through ablations. Similar survival rates alone do not explain why the methods perform similarly.
  • The motivation is unclear: Why should similar performance under this particular configuration inform our understanding of auxiliary rewards more broadly? Overall, the write-up is hard to follow and uses unnecessary jargon.
Questions for authors
  1. Why did you test only one environment? This means you tested only this single environment instead of the auxiliary rewards.
  2. Why did you only test one algorithm? As above, this means you tested only one algorithm instead of the auxiliary rewards.
  3. Why did you only test one algorithm configuration? As above, this basically means you are testing this single configuration instead of the auxiliary rewards.
  4. How did you configure the auxiliary-reward methods? RND, for example needs a learning rate and an optimizer,
1 comment
AuthorsOpenReview ↗

Methodology and Robustness - Response to Reviewer ZB1V

We thank the reviewer for the detailed methodological critique. We agree that the original single-configuration experiment was insufficient to support broad conclusions about auxiliary rewards or PPO mechanisms.

1. Original PPO implementation and clipping (Section 3; Appendix F.5). We clarified that the original study used one full-episode optimizer update rather than conventional minibatch, multi-epoch PPO. The old and current policy probabilities coincide before this update, making active clipping inapplicable as an explanation for the original outcomes. Direct diagnostics confirmed zero objective-active clipping across 30,000 measured original-style updates. We have removed our earlier clipping-based explanation.

2. Conventional PPO robustness (Section 6.2; Appendix K). We implemented a separate rollout-buffer, minibatch, four-epoch PPO pilot covering seven conditions and three paired development seeds. Objective-active clipping occurred in 4,357 of 10,752 optimizer minibatches. However, the baseline survival-learning gate failed, so these results establish optimizer-level differences, not robust performance generalization.

3. Reward scaling and optimization measurements (Section 5; Appendix G). A ten-seed RND intervention used coefficients calibrated on disjoint development data. Increasing RND magnitude changed normalized advantages, common-reference actor gradients, and sampled learned-policy behavior without establishing reliable GridWorld survival improvement.

4. Implementation detail and independent environments. Appendices B–D now specify auxiliary reward definitions, RND architecture and optimizer, PPO configuration, training protocol, and evaluation. We also evaluated official MiniGrid tasks. A focused five-seed FourRooms comparison showed a +12.8-percentage-point observed original-RND effect over PPO, but superiority was not statistically established (Section 6.1; Appendix J). Empty-8x8 provides baseline-learning evidence, not an equivalent auxiliary-reward comparison.

The revised manuscript therefore makes narrower claims: the evaluated auxiliary interventions can alter learning signals and policies without necessarily establishing task improvements. We do not claim generality across algorithms, training budgets, or all PPO configurations.

We would appreciate the reviewer's assessment of whether these corrections and additional experiments address the principal methodological concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Substantive Revisions - Submission #12572

We thank the reviewers for their careful feedback. We have uploaded a revised manuscript and anonymized supplementary archive, and posted individual responses to reviewers ZB1V, Mseu, and 65uq.

The original 40-seed results remain unchanged. The revision makes four substantive contributions:

1. Direct evidence of learning-signal differences. A ten-seed calibrated RND study demonstrates changes in normalized advantages and actor gradients (gradient-influence contrasts: Holm-adjusted p ≈ 0.0296), alongside differences between independently trained policies, without establishing improved GridWorld survival (Section 5; Appendix G).

2. Correction of the original PPO explanation. We verified zero active clipping across 30,000 measured single-pass updates and removed clipping as an explanation for the original outcome similarity. Separate multi-epoch experiments demonstrate active clipping under different configurations (Sections 3, 5.3, and 6.2).

3. Independent-environment evaluation. In a five-seed MiniGrid-FourRooms comparison, original RND achieved an observed +12.8-percentage-point mean success difference over PPO. This result remains statistically inconclusive after correction (Holm p = 0.25; Section 6.1; Appendix J).

4. Clearer scientific contribution and methodological scope. We expanded implementation details, investigated conventional PPO, retained unsuccessful learning-gate results, and reorganized the manuscript around a practical diagnostic distinction between reward influence, optimization changes, learned-policy differences, and task improvement.

The revised paper does not claim universal auxiliary-reward ineffectiveness, performance equivalence, or complete causal explanation. Instead, it demonstrates why similar final scores need not imply identical learning and why measurable optimization differences do not necessarily establish task improvements.

We respectfully ask that these substantive revisions, supporting evidence, and responses to the reviewers' principal concerns be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.