- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Communication Smoothing Distillation (CSD) to transfer communication-enabled MARL policies into decentralized students. It characterizes when a communication policy can be represented using local information, learns cross-agent conditional latent distributions, and regularizes teacher Q-values against communication perturbations before distillation. Evaluation includes a predator–prey diagnostic, SMAC and GRF with restricted visibility, component ablations, and integration with multiple value-decomposition methods.
Critical strengths
-
The paper explicitly addresses information-dependent teacher behavior that decentralized students cannot reproduce. The predator–prey comparison, including a variant without smoothing, supports the practical relevance of this problem and a performance benefit from the smoothing component.
-
The restricted-visibility SMAC and GRF evaluations report competitive aggregate performance, and integration with VDN, QMIX, and Qatten shows that the framework is applicable across several value-decomposition architectures. These results provide meaningful empirical value.
Critical weaknesses
-
The paper defines a joint search over mixture assignments and residual codes, but the code optimizes only residual codes with assignments fixed during the search. Its latent-message scale also includes between-component variance, unlike the paper's weighted component standard deviations; these constructions differ under the soft assignments enabled in the supplied configurations. Additionally, distribution learning uses a reconstruction aggregation and KL surrogate different from the stated variational objective.
-
The proofs establish the existence of a local approximation under uniform smoothness over feasible joint observations. They do not establish that the finite latent search satisfies this assumption or that the optimized student inherits the approximation bound. The teacher–student success-rate gaps do not directly measure policy disagreement or Q variation.
Questions for authors
-
The supplied learner optimizes residual codes while keeping mixture assignments fixed. Was this restricted search used for the results, and how does it correspond to the joint search specified by the method?
-
The code uses a log-sum-exp reconstruction loss, weighted component-wise Gaussian KL terms, and a mixture-balancing penalty. What objective was optimized in the experiments, and what relationship to the stated variational bound is established for this surrogate?
-
What establishes that finite latent perturbations satisfy the theory's uniform smoothness hypothesis and that the learned student, rather than merely an existing local function, satisfies the approximation bound?
-
What reported evidence links the smoothing component's performance benefit specifically to reduced communication-dependent policy or Q discrepancy, rather than to a general regularization effect?
2 comments
Thank you for your response
Although some of the concerns I raised have been addressed, I still believe that the paper essentially meets the minimum acceptance threshold for ICLR. Therefore, I will maintain my original score.
Additional Experiments, Corrections to the Method Description, and Further Analysis of Communication Smoothing
Thank you for recognizing the value of our work. We have added SMAC and GRF ablations and corrected errors in the method description. Our responses follow.
Response to Q1:
Thank you for identifying this issue. Listing both mixture weights and residual codes as search variables was a writing oversight, corrected in Section 4.2.3 and Appendix B.2. The reported experiments perturb only communication samples, consistent with our motivation and submitted code. The GMM weights w, means, and standard deviations define the conditional distribution, while the Sampler generates messages from the sender's actual observation. During perturbation search, distribution parameters and mixture assignments remain fixed; gradient ascent updates only residual codes to maximize teacher Q-value discrepancies.
Response to Q2:
Eq. (8) learns cross-agent conditional observation distributions through reconstruction and latent-distribution regularization. The implementation follows this modeling goal using a regularized GMM surrogate: log-sum-exp aggregates component reconstruction likelihoods; weighted component Gaussian KL provides an analytical upper bound on the mixture KL to the shared prior; and a balancing term encourages component utilization. Related treatments appear in prior work: [1] combines mixture negative log-likelihood with a joint-distribution KL that decomposes into weighted Gaussian KL and mixture-weight KL; [2] adds an entropy regularizer on average auxiliary assignments in a GMM-based generative classifier to encourage balanced assignments. Our implementation draws on these designs, and Appendix B.1 now explains the implementation details.
Response to Q3:
For latent perturbations, we clarify the connection between the maximum-discrepancy objective and Lemma 1. Fix the local information, write the teacher's Q-vector as Qi(ϵ), and let J denote the maximum squared discrepancy in Eq. (14), with the original code ϵ^ as reference. For any ϵ,ϵ′ in the search domain, the triangle inequality gives
\|Q_i(\epsilon)-Q_i(\epsilon')\|_2 \le \|Q_i(\epsilon)-Q_i(\hat\epsilon)\|_2 +\|Q_i(\hat\epsilon)-Q_i(\epsilon')\|_2\\ \le 2\sqrt{J}. \end{aligned}$$ Thus, minimizing $J$ controls pairwise Q variation within this domain, linking the objective to Lemma 1. Finite-step search approximates this maximization. The added iteration study (Figure A7) shows that further search steps bring little additional success-rate improvement within the tested range. For the learned student, Lemma 1 characterizes the existence of a local approximation under its smoothing condition. CSD uses smoothing to approach this condition and Q-value distillation to train a local student to fit the teacher. We assess this training mechanism from two perspectives: 1. **Teacher and student performance is close.** Figure A5 and Table 4 show comparable SMAC success rates; the student's mean success rate is 2.6 percentage points higher than the teacher's. 2. **Ablations in three environments confirm the role of smoothing.** Removing smoothing lowers mean student success rates from 75.2% to 57.4% in Predator-Prey, from 77.6% to 44.0% across the two SMAC tasks, and from 55.8%/61.3% to 47.2%/50.8% on the two GRF tasks. These results support improved distillation and student learning. **Response to Q4 & W2:** We explain the smoothing mechanism and its connection to teacher-student differences from three perspectives: **First, the visualization in Figure A3 illustrates the meaning of communication perturbations.** With agent 2's local observation fixed, gradient search changes the positions of agent 1 and the target encoded in its messages. Smoothing constrains the resulting Q-value differences, reducing dependence on information unavailable to students and producing supervision better suited to CTDE students. **Second, Figure A4 examines communication-dependent Q variation.** We change the position of an agent that has found the target to generate different messages and compare the resulting Q-values. The smoothing loss reduces these differences, improving Q-value consistency across communication inputs. **Third, Figure 2 illustrates the effect on teacher and student performance.** Without smoothing, the teacher retains a near-perfect success rate, but its student's mean success rate is 57.4%. With smoothing, teacher performance decreases in some settings, while student success reaches 75.2%. This supports our design of constraining the communication teacher to improve knowledge transfer to decentralized students. We hope these responses address your concerns and welcome further discussion. [1] Jayaram Raghuram et al. Few-Shot Domain Adaptation for End-to-End Communication. ICLR, 2023. [2] Can Zhang and Gim Hee Lee. GeT: Generative Target Structure Debiasing for Domain Adaptation. ICCV, 2023.