acceptodds
Withdrawn from ICLR 2027

MUTE: Return-Preserving Communication Unlearning for Efficient Multi-Agent Coordination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inter-agent communication is critical for coordinating Multi-Agent Reinforcement Learning (MARL) agents under partial observability to perform effectively in cooperative games; however, real-world bandwidth constraints demand sparse interactions. Prior approaches primarily address this trade-off by optimizing information-theoretic surrogates. We argue that these statistical proxies are fundamentally misaligned with the true objective: a message can be highly informative yet irrelevant to the joint return of the task. In this work, we propose Message Unlearning for Targeted Efficiency (MUTE), a framework that views communication reduction as a value-guided machine unlearning problem. MUTE rigorously quantifies the Counterfactual Message Value using an attention-based estimator, and systematically unlearns the transmission of low-value messages from a policy trained without any communication constraints. This is achieved through a dual-objective mechanism that enforces communication sparsity while preserving the return of the original joint policy. We derive a theoretical upper bound on the performance gap induced by this sparsification, guaranteeing controlled return degradation. We also empirically evaluate MUTE on various complex multi-agent environments, achieving 80% to 90% bandwidth reduction while maintaining performance comparable to state-of-the-art baselines.

Primary area:
reinforcement learning
Keywords:
multi-agent system, reinforcement learning, unlearning, multi-agent communication

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3UWkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUTE reduces communication after training a fully communicative MARL policy. It evaluates messages through counterfactual changes in the joint critic, trains an attention-based message-value estimator, and penalizes low-value messages while anchoring individual Q-values to a frozen teacher. The paper provides a conditional performance bound and evaluates ten scenarios across five environment families.

Critical strengths
  1. Separating cooperative policy learning from communication reduction is well motivated. The training-budget study supports establishing a capable policy before reducing communication.
  2. The experiments cover ten scenarios and report five seeds with confidence intervals. The results suggest promising performance–communication trade-offs.
  3. The appendix includes useful ablations on anchoring, message-value estimation, regularization, and training schedules, alongside gating/penalty controls and computational-cost measurements.
Critical weaknesses
  1. Novelty and baseline coverage remain unclear. CMVC (IEEE TSMC: Systems, 2025) already learns counterfactual message values for selective communication, but is neither discussed nor evaluated. IMGS-MAC (ICRA 2023) and ReSCOM (AAMAS 2025) provide related lossless sparsification and curriculum approaches; both are cited but not compared. The incremental benefit of MUTE's broadcast attribution and anchoring needs clearer justification.

  2. Return preservation is insufficiently justified. As written, Equation 7 feeds the same current messages to teacher and student. If their receiver networks remain identical, the anchor loss stays zero even when removing essential messages changes their original behavior. The teacher's inputs therefore need clarification. Moreover, agreement between learned Q-networks does not establish preserved returns, and greedy discrete actions complicate the theorem's smoothness assumption.

  3. Message magnitude is not a sufficient measure of communication savings. L1 shrinkage does not necessarily reduce transmitted bytes. The send/no-send rule, communication-rate metric, and deployment role of MVE—which consumes all agents' messages—need clarification.

  4. The evaluation does not fully establish selectivity. Dropout robustness can reflect policy adaptation rather than accurate message valuation. Figure 4 appears to compare different final communication rates; matched-rate controls would be more informative. The main-text MMM2 result of 94% also needs reconciliation with 0.86 ± 0.05 in the appendix.

  5. Minor comments. Reconcile λμ_MVE with λ in the algorithm and theorem, and distinguish stopping TD updates from freezing policy parameters.

Questions for authors
  1. How does MUTE differ from CMVC and prior lossless or curriculum-based communication reduction, and can a focused comparison establish the benefit of the proposed combination?
  2. Could you clarify the teacher's message inputs during anchoring and explain how the theoretical assumptions correspond to the implemented policy?
  3. Could you clarify the deployment protocol and communication metric, and strengthen the evidence for selective suppression and preservation at comparable communication rates?
Reviewer_k6DoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies communication efficiency in cooperative multi-agent reinforcement learning.

Existing methods often use measures such as mutual information to evaluate message importance. However, the authors argue that the information contained in a message does not necessarily reflect its contribution to team return.

Therefore, the paper proposes MUTE, which treats communication reduction as a message unlearning problem while preserving task return. The method first trains a multi-agent policy without communication constraints. It then uses CMV to estimate each message's contribution to the joint action-value function. The MVE is used to identify low-value messages. Finally, sparsity loss and behavioral anchoring loss are used to gradually reduce redundant communication.

The authors conduct experiments on 10 task configurations across Hallway, Traffic Junction, SMAC, SMAC-Communication, and SMACv2. The results show that MUTE can greatly reduce the communication rate while maintaining good task performance.

Critical strengths

(1) The research motivation is clear, and the method has practical value.

This paper directly connects message importance with team return. The authors point out that a message may significantly affect other agents' actions without improving team return. I think this is a reasonable observation.

Unlike methods that optimize task performance and communication cost together from the beginning, MUTE first learns a good cooperative policy. It then identifies and reduces redundant communication. This staged approach is straightforward and helps reduce the difficulty of joint optimization.

(2) The experiments are relatively comprehensive and support the main motivation of the method.

The paper compares communication rates and task performance across different cooperative tasks. It also compares MUTE with several existing communication methods.

Figure 2 shows the learning performance and communication rates of different methods. Figure 3 further examines performance under different message dropout rates. Table 2 uses AUC to compare robustness to communication loss.

In addition, the ablation experiments in Figure 4 show that removing behavioral anchoring can cause a clear performance drop. Replacing MVE with random message selection also makes it harder to maintain the original performance. These results provide direct evidence for the roles of the two main components.

The appendix also includes analyses of hyperparameters, training budgets, and computational costs. Overall, the experimental design is relatively complete.

Critical weaknesses

(1) The paper does not clearly explain how the communication rate is calculated or whether its reduction reflects lower communication overhead.

One of the main contributions of this paper is to significantly reduce communication overhead while preserving task performance.

However, the method mainly applies an L1 sparsity penalty to low-value messages, gradually pushing their message vectors toward zero. I did not find a clear explanation of how the Communication Rate is calculated in the experiments. The paper also does not clearly explain how these low-value messages stop being transmitted during execution.

This distinction is important. Reducing the magnitude of message vectors does not necessarily reduce the amount of transmitted data. If agents still send complete zero or near-zero vectors, the communication overhead may not decrease accordingly.

Therefore, I hope the authors can clearly explain how the communication rate is calculated. They should also explain how unnecessary messages are identified and skipped during execution.

(2) The connection between the theoretical performance guarantee and the actual training process is not clear enough.

Theorem 4.2 provides an upper bound on the performance loss after communication unlearning. However, this result assumes bounded MVE estimation error and bounded action-value function error between the original and unlearned policies.

These assumptions are reasonable for theoretical analysis. However, the behavioral anchoring loss used in the paper constrains individual Q-values to preserve policy behavior. In contrast, the theorem directly assumes a bounded error between the joint action-value functions. The paper does not clearly explain how optimizing this loss ensures the condition required by the theorem.

Appendix I reports value function deviations and MVE loss during training. These results help show that the training process is relatively stable. However, average errors or training losses measured on finite samples do not directly establish the uniform error bound required by the theorem.

Therefore, I think the authors should make a clearer distinction between the assumptions used in the theoretical analysis and the properties guaranteed by the training objective. They should also explain how far the experimental results support these assumptions.

Questions for authors

(1) About the actual communication overhead

How exactly is the Communication Rate defined and calculated in this paper? Under what condition is a message considered to be removed?

During execution, if the L1 loss pushes a message vector close to zero, does the agent actually skip the transmission? Or does it still send the complete vector?

I hope the authors can clarify this process. It directly affects whether the reported communication efficiency improvement reflects actual bandwidth savings.

(2) About the theoretical performance guarantee

Theorem 4.2 assumes a bounded error between the joint Q-functions before and after unlearning. However, the behavioral anchoring loss constrains individual Q-values.

Is there a further theoretical connection between these two quantities? In particular, how is this connection established or guaranteed when a nonlinear mixing network is used?

Also, do the experimental results in Appendix I only provide empirical support rather than a strict verification of the theorem's error conditions?

(3) About the implementation of message selection

Eq. (6) uses an adaptive threshold related to the mean MVE prediction to identify messages for unlearning. However, Algorithm 1 in Appendix B directly compares the predicted message value with (\lambda).

Which method is actually used in the implementation? Is the threshold updated dynamically according to the MVE predictions?

Reviewer_teWvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

MUTE proposes to reduce communication in MARL by solving a value-guided machine unlearning problem. It consists of 3 stages. Stage 1 trains an unconstrained communicating policy. Stage 2 trains an attention network, the Message Value Estimator (MVE), to predict each message's Counterfactual Message Value (CMV): the drop in the frozen joint Q-value when that message is masked and teammates' actions are resampled without it. Stage 3 "unlearns": an L1 penalty shrinks messages whose predicted CMV falls below a running-mean threshold, and an anchoring loss keeps each agent's utility Q_i close to the frozen Q*_i. On ten experiment scenarios spanning 4 benchmarks (Hallway, Traffic Junction, SMAC, SMACv2), MUTE reports communication rates of 1–23% with win rates near the full-communication backbone.

Critical strengths

The paper addresses a practically relevant problem: reducing communication overhead without substantially compromising cooperative performance. Evaluating messages by their estimated effect on team return provides an intuitive criterion for identifying redundant communication.

The proposed combination of counterfactual message valuation, an amortized estimator, and behavioral anchoring is conceptually coherent. Predicting all agents’ message values together avoids repeating counterfactual evaluations during each unlearning update.

The empirical evaluation contains useful breadth and controls. Five-seed results span 10 coordination settings in 5 different bencmarks. The appendices include communication-penalty and gating baselines, state-aware estimator ablations, training-time measurements, and budget-allocation experiments.

Critical weaknesses
  1. The advantage of two-stage learning (train without constraint, then unlearn) over joint optimization is not established. Fig. 1 varies fixed communication rates rather than learned gating. Appendix J shows joint training need not collapse but does not establish superiority over a well-tuned penalized gate; the high variance on 1o_10b_vs_1r warrants a penalty sweep.

  2. Relevant one-stage methods that learn efficient communication are missing: NDQ (ICLR 2020), VBC, TMC, SchedNet, ATOC, MBC and Farooq & Iqbal (ICRA 2026). NDQ introduced Hallway and the SMAC-Communication maps and reports over 80% communication reduction without loss, yet neither NDQ nor IMAC is evaluated. IC3Net, the main comparison's explicitly categorized gating baseline, uses a different backbone and does not isolate gating. SMS reaches approximately 55% mean win rate here versus approximately 80% median at 4M in its published experiment on 1o10b_vs_1r. Different statistics and configurations preclude a controlled comparison, but the reproduction settings need clarification.

  3. Closely related staged methods are not discussed. Gated-ACML pretrains with open gates, then learns gates from 1[ΔQ>T]1[\Delta Q>T] using a quantile or EMA threshold. ETCNet also pretrains with full communication before learning a penalized gate. Karten et al. (ICRA 2023), cited only for joint-training difficulties, identify dispensable messages through counterfactual reward tests, reporting 20–75% pruning without retraining and further gate fine-tuning within 10% of total training epochs. Gao et al. (2025) already define "counterfactual message value" with a local predictor. MUTE should explain its distinctions and compare against representative predecessors and a swept-penalty gate on the same backbone.

4 Bandwidth accounting is unspecified. Shrinking a fixed-dimensional, fixed-precision vector does not itself save bits. Yet Fig. 2 plots gate-free TarMAC at fractional rates (0.1–0.9), MUTE's unrestricted pretraining at approximately 0.93–0.98, and MASIA at exactly 1.0. Algorithm 1 specifies attenuation without an explicit transmission mask, whereas the proof assumes masking. Define the communication rate and transmission rule, verify that omitted messages are withheld at execution, and apply consistent accounting across baselines.

  1. Return preservation is overstated. Against the 4M-step full-communication TD baseline in Table 12, MUTE loses 10 and 5 percentage points on 1o_10b_vs_1r and 1o_2r_vs_4r. Strong no-communication results on several maps also motivate a matched-backbone control. Fig. 8 places MUTE below Wnorm=1W_{\mathrm{norm}}=1 on MMM2 and 1o_10b_vs_1r and below zero on Terran, contradicting the blanket preservation claim. Dropout AUC does not isolate selection quality from learned communication independence. The λ\lambda ablation shows threshold sensitivity, while Fig. 4's unequal transmission rates prevent evaluating the MVE's selection advantage at matched communication budgets.

  2. Theorem 4.2 does not establish an actionable guarantee for the implementation. It substitutes the learned critic for true QπQ^\pi, although Phase 3 specifies no policy-evaluation updates or bound on their discrepancy. Term (B)'s uniform ϵ\epsilon is assumed, not certified by per-agent squared anchoring. Likewise, Table 11's average MVE loss does not certify uniform δ\delta, particularly when the frozen estimator encounters altered messages. Evaluate the bound using justified constants and distinguish it from RGMComm's existing return-gap analysis.

References additional to the paper's:

  • Jiang & Lu, "Learning Attentional Communication for Multi-Agent Cooperation," NeurIPS 2018 (ATOC).
  • Kim et al., "Learning to Schedule Communication in Multi-agent Reinforcement Learning," ICLR 2019 (SchedNet).
  • Zhang, Zhang & Lin, "Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control," NeurIPS 2019 (VBC).
  • Zhang, Lin & Zhang, "Succinct and Robust Multi-Agent Communication With Temporal Message Control," NeurIPS 2020 (TMC).
  • Mao et al., "Learning Agent Communication under Limited Bandwidth by Message Pruning," AAAI 2020 (Gated-ACML).
  • Hu et al., "Event-Triggered Communication Network with Limited-Bandwidth Constraint for Multi-Agent Reinforcement Learning," arXiv:2010.04978, 2020 (ETCNet).
  • Han, Dastani & Wang, "Model-based Sparse Communication in Multi-agent Reinforcement Learning," AAMAS 2023 (MBC).
  • Gao et al., "Communication in Multiagent Reinforcement Learning via Counterfactual Message Value," IEEE TSMC: Systems 55(11), 2025.
  • Farooq & Iqbal, "Bandwidth-Efficient Multi-Agent Communication through Information Bottleneck and Vector Quantization," ICRA 2026, arXiv:2602.02035.
Questions for authors

See weakness section. In addition, the anchoring loss as written is degenerate. Is it a typo? Eq. 7 and Algorithm 1 evaluate Qi and Q_i on the same current messages m-i, and Q_i is initialized as a copy of Q_i, so L_anchor is identically zero and, since L_sparse does not depend on θ, θ never moves and MUTE equals "MUTE w/o anchoring".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.