- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Belief over Mean Field (BeMF), a framework for robust MARL under Byzantine attacks, where an unknown subset of agents behaves maliciously.
BeMF uses recurrent neural networks to encode agent behavior histories, estimates agent-level maliciousness probabilities using population-level context, and constructs a belief-weighted mean field for robust policy learning. The method aims to improve scalability by reducing computational complexity as we scale the number agents.
Experiments in Battle (up to 144 agents) and PowerGrid (up to 40 agents) demonstrate improved cooperative rewards over several robust MARL baselines.
Critical strengths
-
Important problem: Addresses robustness against unknown malicious agents in large-scale cooperative MARL.
-
Computational scalability: Replaces expensive pairwise Bayesian beliefs with shared agent-level risk estimates, achieving better scaling.
-
Theoretical foundation: Provides bounds on belief estimation errors, mean-field approximation, and their impact on defensive performance.
Critical weaknesses
-
Limited robustness against adaptive adversaries. Detection performance deteriorates substantially under stealthy attacks. The evaluation does not consider attackers explicitly trained to evade detection while maximizing damage, leaving robustness against adaptive and unseen adversaries uncertain.
-
Incomplete scalability-effectiveness comparison with pairwise-belief methods. The authors demonstrate favorable computational scaling against EIR-MAPPO but do not provide a corresponding comparison of defensive performance.
-
Unverified game-theoretic convergence. Although the paper provides conditional equilibrium approximation guarantees, it does not demonstrate convergence of alternating attacker-defender training or evaluate the exploitability of learned policies against newly optimized attackers.
-
Unclear deployment feasibility. The framework assumes trusted external detectors and population aggregation, but does not evaluate communication overhead, synchronization requirements, or distributed deployment costs.
Questions for authors
-
Architectural contribution: How does BeMF compare with a simpler pipeline consisting of a supervised malicious-agent detector followed by standard mean-field MARL? Can the authors isolate the benefits of individual risk estimation, soft belief weighting, and population-level contextual inference?
-
Adaptive attackers: How does BeMF perform against attackers explicitly trained to maximize cooperative damage while minimizing their detection probability? Does the defense generalize to independently trained or previously unseen adversarial policies?
-
Scalability versus defensive effectiveness: Can the authors compare BeMF against EIR-MAPPO in compatible small-scale settings, measuring both computational cost and defensive effectiveness? What performance trade-off arises from replacing pairwise beliefs with mean-field aggregation?
-
Equilibrium convergence: Does alternating attacker-defender training empirically converge toward a stable equilibrium? Can the authors measure the exploitability of the final defender against a newly trained best-response attacker?
-
Distributed implementation: How are trusted detectors deployed, and how is the belief-weighted mean field communicated to cooperative agents? What are the communication, latency, and synchronization costs as the population grows?
6 comments
Rebuttal by Authors (Part II)
(Continued from Part I)
Agents in a mean-field model interact only through the field, so replacing pairwise beliefs by aggregation costs the finite-population error (Theorem F.1) plus the detection error (Theorem 4.1). Also in our new ablations at 144 agents, show that both compressed pieces carry usable type information. Adding the risk score to a standard mean-field policy improves the return by 246–1369, and adding the belief-weighted field improves it by a further 104–299 (table under “Architectural contribution”).
We add theorem F.3(iii) bounds for idealized alternating updates with a frozen detector, with the infimum taken over attackers near the limit point. Near a nondegenerate local solution of the sup–inf objective, and with enough adversary steps per defender step, it decays linearly with exact gradients and stays at under gradient errors of size .
Equilibrium convergence: Does alternating attacker-defender training empirically converge toward a stable equilibrium? Can the authors measure the exploitability of the final defender against a newly trained best-response attacker?
The results in our paper already use a newly trained attacker. After alternating training, we freeze the final defender and detector and train the attacker against them; Table 1 reports the defender's return under this attacker. We state this protocol explicitly in the revision.
For the final defender and the attacker at the end of alternating training, the attacker-side exploitability is , and it is zero at an equilibrium. We estimate it by , where is the attacker trained against the frozen defender (Battle, ):
| Adv. | Training time attacker-defender reward | Our reported attacker (Table 1) | Relative | |
|---|---|---|---|---|
| 16 | −745.49 ± 16.99 | −783.23 ± 19.79 | 37.74 | 5.1% |
| 32 | −1381.54 ± 26.23 | −1450.11 ± 21.03 | 68.57 | 5.0% |
| 36 | −1538.06 ± 28.35 | −1593.89 ± 22.50 | 55.83 | 3.6% |
| 48 | −2004.95 ± 39.90 | −2162.20 ± 53.74 | 157.25 | 7.8% |
| 72 | −2455.37 ± 56.56 | −2555.94 ± 26.53 | 100.57 | 4.1% |
We find that is positive in all five settings, at 38–157 in return (3.6–7.8%), and exceeds the standard deviation of both columns. The final pair is therefore an approximate equilibrium, not an exact one: further attacker training against the frozen defender finds only a small additional gain, with no collapse. Since is a trained attacker and not a certified best response, is a lower bound on .
Distributed implementation: How are trusted detectors deployed, and how is the belief-weighted mean field communicated to cooperative agents? What are the communication, latency, and synchronization costs as the population grows?
BeMF can run on the monitoring and control channel a power grid already has, with no agent-to-agent communication. Detectors. DERs already report to the operator's control center (SCADA or the microgrid controller running secondary voltage control) and receive set-points from it. The detector runs there as the trusted monitor of Assumption 3.2; a compromised inverter can influence it only through its own behavior. Mean field. The controller appends the DER's score and two 64-dimensional summaries (516 bytes) to the reference signal it already sends every cycle. About the cost, see the table belew for measured payloads and the step time at 288 agents. We add this table to our revised papar at Appendix A.3 Table 4
| Quantity | Per agent | All protected agents | Growth in |
|---|---|---|---|
| Uplink telemetry per step | ≈8.95 kB | 1,289,233 bytes | per agent, total |
| Downlink per step, unicast | 516 bytes | 74,304 bytes | per agent, total |
| Downlink per step, ideal broadcast | 516 bytes | 1,088 bytes | 512 bytes fixed + 4 bytes per agent |
| Message rounds per step | 1 up, 1 down | 1 up, 1 down | independent of |
Per step each agent sends ≈8.95 kB and receives 516 bytes, independent of , in one message round. A full monitor pass takes 8.90 ms for 288 agents, and an 11× larger population (32 → 364 agents) takes 2.8× the time per episode (Table 5 in our paper). Timings exclude network transport.
Invitation to Author- Reviewer Discussion
Dear Reviewer,
We are the authors of Submission 5649: "Robust Large-Scale Multi-agent Reinforcement Learning against Byzantine Fault". We’ve uploaded a point-to-point response to your comments, which we believe addresses your concerns.
As the Author / Reviewer discussion phase is going to end, we would greatly appreciate it if you could take a moment to share your thoughts on our rebuttal. Your feedback is valuable to us, and we’d be happy to further clarify any remaining points if needed.
Best regards,
Authors of Submission 5649
Rebuttal by Authors (Part I)
We thank Reviewer kLJb for the constructive review and address each point below.
BeMF also uses a supervised detector; the difference is how its output enters control. The closest variant to the pipeline you describe is w/o weighted field: the detector's risk score plus a standard, unweighted mean field. We run a full ablation study, each ablation removes one component (Battle, 144 agents, mean ± std):
We can see that without type information (w/o belief) the policy cannot separate cooperative from Byzantine behavior (Prop. 4.2); it is worst in four of five settings. Soft weighting. w/o weighted field aggregates with the raw mean field, whose distortion the attacker controls; weighting by 1−pi bounds it by the detection error (Prop. 4.1). Mean-field summaries. w/o mean field keeps only the score and leaves no population signal to coordinate on (184–554 below BeMF). Population context. Masking the pooled context ζtN fed to the detector (w/o pop. context) prevents it from estimating the compromised fraction (68–227 below BeMF).
We design two kinds of adaptive and unseen attacker: (1) Intimate attacker: Byzantine agents follow the cooperative policy for the first third of the episode and attack afterwards. (2) Random selection attacker: Replacing VAI selection with uniformly random selection.
BeMF holds against adaptive and unseen attacker we tested, and the objective you describe is already contained in the attackers we train. The defender and detector are frozen in all tests. An imitative attacker that first cooperates to obtain a low pi yields a higher return in all nine settings (+1.8% to +21.1%); random selection in place of VAI yields a higher return in four of five settings (lower within one std at 16 adversaries). Evasion also costs damage: when the attacker is diluted toward the cooperative policy (Table 12), detection falls to chance only at λ≥0.8, where it no longer reduces the team return. We put the full data in Table 9-10 of our revised pdf.
From Table 3 in our original upload paper (Table 6 now), we can see that in our Battle setup EIR-MAPPO needs 7.8 GPU-days per run at 16 agents against 0.26 for BeMF (30×), and 45× and 74× more at 32 and 64 agents. PowerGrid Env costs more time to run. A small-scale return comparison would not isolate the defense. The backbones differ (MAPPO with identity-indexed beliefs vs. mean-field actor–critic), and EIR-MAPPO's environments have at most four agents and one adversary, where a mean-field method is dominated by its O(N−1/2) approximation error. In our environments one EIR-MAPPO seed at the smallest population takes 14.6 GPU-days.
(Continued in Part II below)