- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper performs a controlled experiment to examine whether different bonus rewards can produce similar results when used with the same reinforcement learning algorithm. It compares six versions of PPO in a survival game, with bonuses encouraging different behaviours. The paper shows that although encouraged to act different - the outcomes of all the versions is roughly the same 33% survival rate.
Critical strengths
The controlled comparison across 40 matched seeds is a substantial. Holding the PPO architecture and training configuration fixed, together with evaluation on held-out maps, makes the comparison more reliable.
Critical weaknesses
-
The comparison is restricted to one environment, one training budget, and fixed bonus coefficients. The authors also acknowledge that the curiosity bonuses are tiny relative to the environmental reward. Consequently, the similar performance may depend on this particular configuration.
-
The paper describes how PPO could suppress differences between auxiliary rewards, but does not measure the advantages, clipping activity, or gradients needed to investigate those mechanisms. Similar final survival rates cannot distinguish similar updates from different learning processes that reach similar outcomes. Although the authors acknowledge this limitation, it leaves the central explanatory contribution unsupported by direct empirical evidence.
Questions for authors
-
Can the authors demonstrate a similar effect on more environments?
-
Can the authors suggest a concrete change to the reward design or PPO training process that could make the auxiliary rewards more effective in this setting? Which proposed explanation for the similar outcomes would this change address?
-
Could you provide a small coefficient or bonus-normalization ablation? For the recorded bonuses which are very small relative to the task-reward terms and are further reduced by their coefficients.
Reward Calibration and Additional Experiments - Response to Reviewer 65uq
We thank the reviewer for requesting reward-scale calibration, independent-environment evaluation, and a more concrete investigation of auxiliary-reward effectiveness. We have revised the manuscript and conducted additional experiments addressing these points.
1. Reward-scale intervention (Section 5.1; Appendix G, Tables 12–14). We calibrated three RND multipliers on separate development seeds and evaluated them over ten paired training seeds. Increasing RND strength produced measurable differences in normalized advantages and actor gradients (gradient-influence contrasts: Holm-adjusted p ≈ 0.0296). However, the observed GridWorld survival changes were −0.48, −2.00, and −1.47 percentage points relative to original RND, with no established improvement.
This provides a concrete intervention and an important qualification: increasing an initially weak auxiliary reward changes the learning signal, but stronger influence does not necessarily improve task performance. Measuring achieved reward strength and actor-gradient influence can therefore help distinguish ineffective scaling from a lack of demonstrated downstream benefit.
2. Independent-environment evaluation (Section 6.1; Appendix J, Table 24). In MiniGrid-FourRooms, original-scale RND achieved 53.6% mean success versus 40.8% for PPO across five paired development seeds (+12.8 percentage points). This observed gain did not satisfy the prespecified exact-test superiority rule (Holm-adjusted p = 0.25), and we do not claim general superiority.
3. PPO mechanism correction (Section 5.3; Appendix F.5). We verified that active clipping was absent across 30,000 measured original-style updates and removed clipping as an explanation for the original results.
The revised paper now distinguishes reward magnitude, optimization-level influence, learned-policy differences, and task outcomes, while acknowledging unresolved generalization.
We would appreciate the reviewer's assessment of whether these additional experiments and corrected interpretations address the principal concerns.