- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
They modify PPO’s surrogate so that ratios moving well past 1 in the “wrong” direction, against the sign of the advantage, get a steeper slope, which effectively increases the gradient for those samples. This is supported with a one step lemma showing the update keeps such ratios closer to 1 than PPO, a tabular setting convergence proof, and comparisons against PPO and variants on mujoco+atari.
Critical strengths
- The idea is simple, cheap, and can be drop-in. It’s a few lines of code with no extra compute. and one fixed setting (α = 3, β = ε) is used everywhere. if the gains hold up, people would actually use it.
- The Atari evaluation is reasonably broad, and the authors use appropriate quantificaiton( stratified bootstrap CIs).
Critical weaknesses
No major weaknesses that would prevent adoption, (given that results hold for other researchers).
Questions for authors
Why do Tables 3 and 7 disagree for nominally identical runs? which is correct?
We thank the reviewer for his positive feedback. Below we address the reviewer's question.
Questions
Why do Tables 3 and 7 disagree for nominally identical runs? which is correct?
Table 3 shows the performance comparisons on MuJoCo using 10 seeds whereas Table 7 shows the same performance comparison on 5 seeds when we compare all variants considered in the paper to DC-RB and DC-Leaky (each evaluated over the same 5 seeds).