- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents that mutual distillation between reinforcement learning policies serves as implicit regularization that prevents them from overfitting to irrelevant features. It establishes a trust-region lower bound connecting generalization performance with policy robustness. Experiment results demonstrate that MDPO improves generalization and visual robustness on Procgen.
Critical strengths
-
The paper provides useful theoretical insights, establishing a trust-region lower bound connecting generalization performance with policy robustness. It further characterizes the effect of mutual distillation under ideal same-view updates and establishes sufficient conditions for robustness improvement under approximate updates. It also shows that same-view agreement alone is insufficient.
-
The experimental results support that MDPO’s generalization gains are related to the representations learned by the encoder, providing empirical support for the paper’s claim.
Critical weaknesses
-
Theorem 3.14 analyzes a weighted mixture of the action probabilities output by the policies. Theorem 3.15 extends this analysis to actual approximate updates and provides conditions under which robustness can still improve despite deviations from the ideal mixture. MDPO uses PPO and KL losses for training, with the KL term implementing mutual policy distillation. However, the paper does not explain why this practical training approach satisfies the conditions established in Theorem 3.15 for robustness improvement under actual updates. This limits the theory’s ability to explain the experimental results.
-
The authors argue that mutual distillation between two policies can cancel complementary rendering errors, thereby improving policy robustness. While I can see from the experimental results that robustness improves, the experiments do not directly demonstrate that complementary rendering errors are indeed canceled during actual training. Therefore, the existing experimental results are insufficient to support the mechanism proposed in the paper.
Questions for authors
-
How do the PPO and KL gradient updates used by MDPO relate to the action-probability mixing in Theorem 3.14? Can the authors also provide theoretical or empirical evidence demonstrating that MDPO satisfies the conditions required by Theorem 3.15 for approximate updates to improve robustness?
-
Can the authors provide direct evidence that complementary rendering errors are indeed canceled during actual MDPO training? Furthermore, can the robustness improvements resulting from this cancellation be separated from the effects of other factors to assess whether complementary rendering error cancellation is indeed an important cause of MDPO’s improved robustness?
I would consider increasing my score if the authors can convincingly address these concerns.