acceptodds
Under review as a conference paper at ICLR 2027

Towards a Theoretical Understanding of How Mutual Distillation Improves Generalization

Abstract

We argue that mutual distillation between reinforcement learning policies serves as implicit regularization that prevents them from overfitting to irrelevant features. Theoretically, we establish the first trust region lower bound connecting generalization performance with policy robustness, confirming the long-standing assumption that policy robustness underpins generalization. We further characterize the missing connection between deep mutual learning (DML) and this lower bound. Empirically, we demonstrate that mutual distillation between policies promotes such robustness, enabling the spontaneous emergence of invariant representations from visual inputs. Ultimately, rather than emphasizing extensive baseline comparisons, our goal is to uncover the underlying principles of generalization and provide theoretical insights into its mechanisms as our foundational contributions.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Gradient, Generalization Theory, Representation Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3gvGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents that mutual distillation between reinforcement learning policies serves as implicit regularization that prevents them from overfitting to irrelevant features. It establishes a trust-region lower bound connecting generalization performance with policy robustness. Experiment results demonstrate that MDPO improves generalization and visual robustness on Procgen.

Critical strengths
  1. The paper provides useful theoretical insights, establishing a trust-region lower bound connecting generalization performance with policy robustness. It further characterizes the effect of mutual distillation under ideal same-view updates and establishes sufficient conditions for robustness improvement under approximate updates. It also shows that same-view agreement alone is insufficient.

  2. The experimental results support that MDPO’s generalization gains are related to the representations learned by the encoder, providing empirical support for the paper’s claim.

Critical weaknesses
  1. Theorem 3.14 analyzes a weighted mixture of the action probabilities output by the policies. Theorem 3.15 extends this analysis to actual approximate updates and provides conditions under which robustness can still improve despite deviations from the ideal mixture. MDPO uses PPO and KL losses for training, with the KL term implementing mutual policy distillation. However, the paper does not explain why this practical training approach satisfies the conditions established in Theorem 3.15 for robustness improvement under actual updates. This limits the theory’s ability to explain the experimental results.

  2. The authors argue that mutual distillation between two policies can cancel complementary rendering errors, thereby improving policy robustness. While I can see from the experimental results that robustness improves, the experiments do not directly demonstrate that complementary rendering errors are indeed canceled during actual training. Therefore, the existing experimental results are insufficient to support the mechanism proposed in the paper.

Questions for authors
  1. How do the PPO and KL gradient updates used by MDPO relate to the action-probability mixing in Theorem 3.14? Can the authors also provide theoretical or empirical evidence demonstrating that MDPO satisfies the conditions required by Theorem 3.15 for approximate updates to improve robustness?

  2. Can the authors provide direct evidence that complementary rendering errors are indeed canceled during actual MDPO training? Furthermore, can the robustness improvements resulting from this cancellation be separated from the effects of other factors to assess whether complementary rendering error cancellation is indeed an important cause of MDPO’s improved robustness?

I would consider increasing my score if the authors can convincingly address these concerns.

Reviewer_nAiJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why mutual distillation improves zero-shot generalization in visual RL. Theoretically, the authors establish a trust-region lower bound connecting generalization performance to policy robustness against rendering perturbations. They prove that Deep Mutual Learning (DML) transfers robustness, cancels complementary errors via policy mixing, and guarantees robustness gains under bounded update residuals. Empirically, they propose MDPO, which concurrently trains two PPO agents using mutual KL regularization without data augmentations. On Procgen (16 environments), MDPO consistently outperforms PPO. Controlled ablations—including a 2times compute/parameter baseline and linear probes—verify that the gains stem from learning invariant representations.

Critical strengths
  1. The theoretical analysis is careful and nontrivial. In particular, the paper shows that peer agreement alone cannot guarantee robustness, and then gives explicit conditions under which mutual distillation can help.
  2. The experiments go beyond final return. The robustness test, encoder transfer, KL ablation, and compute-matched PPO baseline provide useful evidence for the proposed explanation.
Critical weaknesses
  1. The main theoretical gap is that the robustness-improvement result analyzes an idealized policy mixture, not the actual neural update used by MDPO. The approximation residual is introduced theoretically but not measured in experiments.
  2. The theory assumes pure rendering changes over the same underlying MDP, while the main Procgen evaluation uses unseen levels. The relationship between these two settings is not fully clear.
Questions for authors
  1. Can the authors measure how closely the actual MDPO update matches the idealized update in the theory, for example by estimating the residual term in Theorem 3.15?
  2. How should the Procgen unseen-level results be interpreted relative to the theoretical assumption of pure rendering changes?
  3. Is the proposed lower bound quantitatively informative in practice, or is it mainly intended as a qualitative explanation?
Reviewer_GFFpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks why training two reinforcement learning policies with a mutual distillation loss makes them generalize across bijective rendering functions. The theory assumes one underlying MDP, and the states of this MDP are only seen by the agents through observations by the rendering functions. The paper gives a lower bound on the return under all rendering functions in terms of the policy's robustness to the rendering (RπR_\pi). The authors then argue that mutual distillation reduces RπR_\pi. The proposed method, MDPO, jointly trains two PPO agents on the same levels of Procgen environments and adds a KL term between their action distributions. The experiments treat each level as one rendering function. MDPO is compared with different PPO variants. On the test levels, MDPO's return is higher than PPO's in about ten environments. The authors measure the robustness of the trained policies via a proxy of RπR_\pi and t-SNE plots.

Critical strengths
  1. I checked the mathematical details and they appear to be correct albeit difficult to read at times (such as the statement of Theorem 3.4).

  2. The doubled PPO baseline gives a fair comparison regarding cost and - in the four environments where it is tested - MDPO reaches a higher return.

Critical weaknesses
  1. From the title, a reader would expect an explanation of how mutual distillation improves generalization. This would translate into showing that MDPO's training decreases RπR_\pi, which is not shown. Theorem 3.13 shows robustness transfers from one agent to the other, so it needs one agent to be robust already. In MDPO such an agent is not provided, as they both start from random initialization and train on the same levels. (Table 1 shows that neither agent ends up robust.) Theorem 3.14 could achieve a decrease in RπR_\pi via averaging, but MDPO does not average action distributions. MDPO takes gradient steps on the KL term, and Theorem 3.15 connects these steps to an averaged policy through a residual that is not bounded. The authors show in Appendix E.8 that this can fail by providing an example where a gradient step lowers the KL term and raises RπR_\pi for both agents. Putting this aside and accepting the residual, the paper also argues using Corollary 3.16 that a smaller RπR_\pi gives better generalization (let us denote this by Argument A). Robustness appears in Eq. (20) through BiB_i in the term Cmin⁡{1,Bi/(1−γ)}C\min\{1, B_i/(1-\gamma)\}, which only depends on BiB_i when Bi<1−γB_i < 1 - \gamma. For the paper's γ=0.999\gamma = 0.999, this means Bi<0.001B_i < 0.001. But by Theorem 3.15, Bi≥Rπ~iB_i \ge R_{\tilde\pi_i}, and Rπ~iR_{\tilde\pi_i} is at least the proxy R^\hat R. Table 1 gives R^≥0.43\hat R \ge 0.43 for every policy, so the term is the constant CC and Argument A does not apply.

  2. The theory targets the gap between the return on the training levels and on the test levels. The experiments do not show a smaller gap for MDPO than for PPO. MDPO raises both by similar amounts (Figs. 7 and 8). The large gains of MDPO are on bigfish, dodgeball, fruitbot and starpilot. In the first three, PPO's return on the training levels stays near its initial value for the whole run, so the comparison is between a policy that learned and one that did not. . In the other twelve environments the gains are smaller. In heist, jumper and maze no improvement is seen. Therefore, the claim in Section 5.2 that MDPO outperforms PPO in all environments is not consistent with Fig. 7. Finally, the measured R^\hat R does not correlate with the gains on the environments in Table 1: the two largest improvements in R^\hat R are on heist and maze, which display no improvement upon PPO. Meanwhile, starpilot, with the largest gain of the eight, has the smallest improvement.

  3. The experiments that analyze where the performance improvements compared to PPO come from are run (with one exception) in the environments where PPO has not learned. This makes it hard to separate the paper's explanation for the improvements from the simple one that MDPO starts learning earlier. Therefore, the t-SNE plots and the linear probes compare a trained encoder with an untrained one.

Questions for authors

Is R^\hat R of the averaged policy smaller than R^\hat R of either agent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.