acceptodds
Under review as a conference paper at ICLR 2027

Robust Large-Scale Multi-agent Reinforcement Learning against Byzantine Fault

Abstract

In a large-scale multi-agent system, several corrupted agents with unknown identities can derail the system by spreading abnormal behaviors that harm coordination. Such failures are known as Byzantine faults, under which defenders know neither the number nor the identities of agents being attacked. The classic solution concept is a Bayesian equilibrium, where defenders act on beliefs about whether each agent is cooperative or adversarial. However, this inference becomes intractable as the agent population grows, since the number of possible corrupted subsets increases exponentially. Our key insight is that individual defenders do not need to identify the type of the whole team, but only to aggregate their type uncertainty into the mean field, capturing how the population is distorted rather than by whom. Here we propose Belief over Mean Field (BeMF), which realizes this as robust mean-field control over two agent types. BeMF assigns each agent a Byzantine risk score in linear time, uses these scores to build a belief-weighted mean field, and learns a shared policy that acts on it. We prove that BeMF's belief-weighted mean-field model approximates both population behavior and worst-case performance with explicit error bounds that shrink as the population grows, so the attacker–defender equilibrium remains approximately valid at finite population sizes. We evaluate BeMF on red-versus-blue team combat with up to 144 agents and power-grid control with 20 to 40 agents. In both domains, BeMF outperforms representative baselines even when half of the agents are corrupted. These results establish BeMF as a scalable approach to robust multi-agent cooperation.

Primary area:
reinforcement learning
Keywords:
large-scale, multi-agent, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kLJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Belief over Mean Field (BeMF), a framework for robust MARL under Byzantine attacks, where an unknown subset of agents behaves maliciously.

BeMF uses recurrent neural networks to encode agent behavior histories, estimates agent-level maliciousness probabilities using population-level context, and constructs a belief-weighted mean field for robust policy learning. The method aims to improve scalability by reducing computational complexity as we scale the number agents.

Experiments in Battle (up to 144 agents) and PowerGrid (up to 40 agents) demonstrate improved cooperative rewards over several robust MARL baselines.

Critical strengths
  • Important problem: Addresses robustness against unknown malicious agents in large-scale cooperative MARL.

  • Computational scalability: Replaces expensive pairwise Bayesian beliefs with shared agent-level risk estimates, achieving better scaling.

  • Theoretical foundation: Provides bounds on belief estimation errors, mean-field approximation, and their impact on defensive performance.

Critical weaknesses
  • Limited robustness against adaptive adversaries. Detection performance deteriorates substantially under stealthy attacks. The evaluation does not consider attackers explicitly trained to evade detection while maximizing damage, leaving robustness against adaptive and unseen adversaries uncertain.

  • Incomplete scalability-effectiveness comparison with pairwise-belief methods. The authors demonstrate favorable computational scaling against EIR-MAPPO but do not provide a corresponding comparison of defensive performance.

  • Unverified game-theoretic convergence. Although the paper provides conditional equilibrium approximation guarantees, it does not demonstrate convergence of alternating attacker-defender training or evaluate the exploitability of learned policies against newly optimized attackers.

  • Unclear deployment feasibility. The framework assumes trusted external detectors and population aggregation, but does not evaluate communication overhead, synchronization requirements, or distributed deployment costs.

Questions for authors
  • Architectural contribution: How does BeMF compare with a simpler pipeline consisting of a supervised malicious-agent detector followed by standard mean-field MARL? Can the authors isolate the benefits of individual risk estimation, soft belief weighting, and population-level contextual inference?

  • Adaptive attackers: How does BeMF perform against attackers explicitly trained to maximize cooperative damage while minimizing their detection probability? Does the defense generalize to independently trained or previously unseen adversarial policies?

  • Scalability versus defensive effectiveness: Can the authors compare BeMF against EIR-MAPPO in compatible small-scale settings, measuring both computational cost and defensive effectiveness? What performance trade-off arises from replacing pairwise beliefs with mean-field aggregation?

  • Equilibrium convergence: Does alternating attacker-defender training empirically converge toward a stable equilibrium? Can the authors measure the exploitability of the final defender against a newly trained best-response attacker?

  • Distributed implementation: How are trusted detectors deployed, and how is the belief-weighted mean field communicated to cooperative agents? What are the communication, latency, and synchronization costs as the population grows?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part I)

We thank Reviewer kLJb for the constructive review and address each point below.

Architectural contribution: How does BeMF compare with a simpler pipeline consisting of a supervised malicious-agent detector followed by standard mean-field MARL? Can the authors isolate the benefits of individual risk estimation, soft belief weighting, and population-level contextual inference?

BeMF also uses a supervised detector; the difference is how its output enters control. The closest variant to the pipeline you describe is w/o weighted field: the detector's risk score plus a standard, unweighted mean field. We run a full ablation study, each ablation removes one component (Battle, 144 agents, mean ± std):

Adv. numw/o beliefw/o mean fieldw/o weighted fieldw/o population contextBeMF
16-2450.65 ± 122.52-1337.68 ± 23.10-1081.99 ± 18.77-947.46 ± 8.38-783.23 ± 19.79
32-2739.41 ± 122.58-1786.58 ± 34.22-1568.15 ± 29.12-1530.59 ± 19.74-1450.11 ± 21.03
36-2809.25 ± 74.69-1871.95 ± 33.70-1787.78 ± 41.05-1820.85 ± 11.26-1593.89 ± 22.50
48-2569.90 ± 116.71-2576.57 ± 68.91-2321.13 ± 51.80-2376.76 ± 19.67-2162.20 ± 53.74
72-2905.39 ± 101.47-2739.61 ± 42.15-2659.62 ± 38.76-2624.37 ± 20.68-2555.94 ± 26.53

We can see that without type information (w/o belief) the policy cannot separate cooperative from Byzantine behavior (Prop. 4.2); it is worst in four of five settings. Soft weighting. w/o weighted field aggregates with the raw mean field, whose distortion the attacker controls; weighting by 1−pi1-p_i bounds it by the detection error (Prop. 4.1). Mean-field summaries. w/o mean field keeps only the score and leaves no population signal to coordinate on (184–554 below BeMF). Population context. Masking the pooled context ζtN\zeta_t^N fed to the detector (w/o pop. context) prevents it from estimating the compromised fraction (68–227 below BeMF).

Adaptive attackers: How does BeMF perform against attackers explicitly trained to maximize cooperative damage while minimizing their detection probability? Does the defense generalize to independently trained or previously unseen adversarial policies?

We design two kinds of adaptive and unseen attacker: (1) Intimate attacker: Byzantine agents follow the cooperative policy for the first third of the episode and attack afterwards. (2) Random selection attacker: Replacing VAI selection with uniformly random selection.

Adv. numVAI selection (Table 1)Random selectionIntimate attacker
16−783.23 ± 19.79−1006.72 ± 253.11−755.41 ± 8.55
32−1450.11 ± 21.03−1264.43 ± 241.02−1382.98 ± 10.26
36−1593.89 ± 22.50−1553.81 ± 192.37−1551.57 ± 17.47
48−2162.20 ± 53.74−1742.35 ± 158.90−1801.91 ± 24.98
72−2555.94 ± 26.53−2360.97 ± 167.56−2132.11 ± 6.30

BeMF holds against adaptive and unseen attacker we tested, and the objective you describe is already contained in the attackers we train. The defender and detector are frozen in all tests. An imitative attacker that first cooperates to obtain a low pip_i yields a higher return in all nine settings (+1.8% to +21.1%); random selection in place of VAI yields a higher return in four of five settings (lower within one std at 16 adversaries). Evasion also costs damage: when the attacker is diluted toward the cooperative policy (Table 12), detection falls to chance only at λ≥0.8\lambda\ge0.8, where it no longer reduces the team return. We put the full data in Table 9-10 of our revised pdf.

Scalability versus defensive effectiveness: Can the authors compare BeMF against EIR-MAPPO in compatible small-scale settings, measuring both computational cost and defensive effectiveness? What performance trade-off arises from replacing pairwise beliefs with mean-field aggregation?

From Table 3 in our original upload paper (Table 6 now), we can see that in our Battle setup EIR-MAPPO needs 7.8 GPU-days per run at 16 agents against 0.26 for BeMF (30×), and 45× and 74× more at 32 and 64 agents. PowerGrid Env costs more time to run. A small-scale return comparison would not isolate the defense. The backbones differ (MAPPO with identity-indexed beliefs vs. mean-field actor–critic), and EIR-MAPPO's environments have at most four agents and one adversary, where a mean-field method is dominated by its O(N−1/2)O(N^{-1/2}) approximation error. In our environments one EIR-MAPPO seed at the smallest population takes 14.6 GPU-days.

(Continued in Part II below)

AuthorsOpenReview ↗

Rebuttal by Authors (Part II)

(Continued from Part I)

Agents in a mean-field model interact only through the field, so replacing pairwise beliefs by aggregation costs the finite-population error (Theorem F.1) plus the detection error (Theorem 4.1). Also in our new ablations at 144 agents, show that both compressed pieces carry usable type information. Adding the risk score to a standard mean-field policy improves the return by 246–1369, and adding the belief-weighted field improves it by a further 104–299 (table under “Architectural contribution”).

We add theorem F.3(iii) bounds δatt\delta_{\mathrm{att}} for idealized alternating updates with a frozen detector, with the infimum taken over attackers near the limit point. Near a nondegenerate local solution of the sup–inf objective, and with enough adversary steps per defender step, it decays linearly with exact gradients and stays at O(εg2)O(\varepsilon_{\mathrm g}^2) under gradient errors of size εg\varepsilon_{\mathrm g}.

Equilibrium convergence: Does alternating attacker-defender training empirically converge toward a stable equilibrium? Can the authors measure the exploitability of the final defender against a newly trained best-response attacker?

The results in our paper already use a newly trained attacker. After alternating training, we freeze the final defender and detector and train the attacker against them; Table 1 reports the defender's return under this attacker. We state this protocol explicitly in the revision.

For the final defender π\pi and the attacker ω\omega at the end of alternating training, the attacker-side exploitability is δatt=J(π,ω)−inf⁡ω′J(π,ω′)≥0\delta_{\mathrm{att}} = J(\pi,\omega) - \inf_{\omega'} J(\pi,\omega') \ge 0, and it is zero at an equilibrium. We estimate it by δ^att=J^(π,ω)−J^(π,ωbr)\widehat\delta_{\mathrm{att}} = \widehat J(\pi,\omega) - \widehat J(\pi,\omega^{\mathrm{br}}), where ωbr\omega^{\mathrm{br}} is the attacker trained against the frozen defender (Battle, N=144N=144):

Adv.Training time attacker-defender rewardOur reported attacker (Table 1)δ^att\widehat\delta_{\mathrm{att}}Relative
16−745.49 ± 16.99−783.23 ± 19.7937.745.1%
32−1381.54 ± 26.23−1450.11 ± 21.0368.575.0%
36−1538.06 ± 28.35−1593.89 ± 22.5055.833.6%
48−2004.95 ± 39.90−2162.20 ± 53.74157.257.8%
72−2455.37 ± 56.56−2555.94 ± 26.53100.574.1%

We find that δ^att\widehat\delta_{\mathrm{att}} is positive in all five settings, at 38–157 in return (3.6–7.8%), and exceeds the standard deviation of both columns. The final pair is therefore an approximate equilibrium, not an exact one: further attacker training against the frozen defender finds only a small additional gain, with no collapse. Since ωbr\omega^{\mathrm{br}} is a trained attacker and not a certified best response, δ^att\widehat\delta_{\mathrm{att}} is a lower bound on δatt\delta_{\mathrm{att}}.

Distributed implementation: How are trusted detectors deployed, and how is the belief-weighted mean field communicated to cooperative agents? What are the communication, latency, and synchronization costs as the population grows?

BeMF can run on the monitoring and control channel a power grid already has, with no agent-to-agent communication. Detectors. DERs already report to the operator's control center (SCADA or the microgrid controller running secondary voltage control) and receive set-points from it. The detector runs there as the trusted monitor of Assumption 3.2; a compromised inverter can influence it only through its own behavior. Mean field. The controller appends the DER's score and two 64-dimensional summaries (516 bytes) to the reference signal it already sends every cycle. About the cost, see the table belew for measured payloads and the step time at 288 agents. We add this table to our revised papar at Appendix A.3 Table 4

QuantityPer agentAll protected agentsGrowth in NN
Uplink telemetry per step≈8.95 kB1,289,233 bytesO(1)O(1) per agent, O(N)O(N) total
Downlink per step, unicast516 bytes74,304 bytesO(1)O(1) per agent, O(N)O(N) total
Downlink per step, ideal broadcast516 bytes1,088 bytes512 bytes fixed + 4 bytes per agent
Message rounds per step1 up, 1 down1 up, 1 downindependent of NN

Per step each agent sends ≈8.95 kB and receives 516 bytes, independent of NN, in one message round. A full monitor pass takes 8.90 ms for 288 agents, and an 11× larger population (32 → 364 agents) takes 2.8× the time per episode (Table 5 in our paper). Timings exclude network transport.

AuthorsOpenReview ↗

Invitation to Author- Reviewer Discussion

Dear Reviewer, 

We are the authors of Submission 5649: "Robust Large-Scale Multi-agent Reinforcement Learning against Byzantine Fault". We’ve uploaded a point-to-point response to your comments, which we believe addresses your concerns. 

As the Author / Reviewer discussion phase is going to end, we would greatly appreciate it if you could take a moment to share your thoughts on our rebuttal. Your feedback is valuable to us, and we’d be happy to further clarify any remaining points if needed.

Best regards,

Authors of Submission 5649

Reviewer_RgitOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Byzantine faults at scale in cooperative MARL, where the number and identities of corrupted agents are unknown and each compromised agent can be controlled arbitrarily but can influence the system only through its own actions. The classical Bayesian-equilibrium solution requires maintaining a posterior over 2^N controlled subsets and does not scale. The key insight is that, under mean-field interaction, the defender does not need to identify the type of the entire team and it suffices to aggregate type uncertainty into the mean field and characterize the population-level distortion rather than 'who is corrupted'. Therefore, the paper proposes the BeMF algorithm, which (i) uses a shared detector to estimate a Byzantine risk for each agent from its local history and a population-level prior, with another network estimating the attack fraction; (ii) replaces the raw mean field with a belief-weighted mean field; and (iii) trains a shared defender policy against a worst-case attacker. Theoretically, this paper establishes belief-dependent performance bounds, mean-field approximation guarantees, and the transfer of approximate saddles to finite populations. Experiments in Battle and PowerGrid show that BeMF is best in 16 of 18 configurations, compared with MTMFQ, MIR3, RoMFAC, and RMFTG.

Critical strengths

(1) The core insight and the scalable architecture. Instead of enumerating 2^N controlled subsets or maintaining an O(N^2) observer-target belief, BeMF uses a shared detector plus a belief-weighted mean field to compress type uncertainty into a population-level distortion. As a result, the inference is O(N), network size is independent of N, and deployment requires neither attacker identities nor the attack fraction. This is a key step in moving Bayesian equilibrium from 'small-scale, single attacker' to 'large-scale, unknown fraction'. The experiments remain effective at 144 agents and up to 50% corruption, showing that this is not a toy. (2) The theoretical analysis is clear and easy to follow. The paper moves from field stability (Prop. 4.1) to a closed-loop finite-horizon gap (Thm 4.1), detector error decomposition (Prop. E.1), nested-information ordering (Prop. 4.2), and mean-field limit/finite-population saddle transfer (App. F), while explicitly distinguishing the contraction for exact policy evaluation from the optimization residual of the actual PPO. It lets the reader know exactly what the guarantee covers and does not cover.

Critical weaknesses

(1) The 'Byzantine robustness' guarantee is average-case/detection-based, not a classical worst-case/breakdown guarantee. The belief-weighted field is a linear weighted average, thus if the detector assigns a very low p_i to an attacker (false negative), that agent enters the mean field with weight close to one and can substantially pollute it. Classical Byzantine-robust aggregation emphasizes the breakdown point precisely because it targets the worst case rather than the average. Prop. E.1 also acknowledges that e_t cannot be small when the two behavior classes overlap heavily. In addition, the detector is supervised with ground-truth type labels, and the framework assumes an external monitor that broadcasts p_i and is not controlled by the adversary, requiring labeled data at training time and a trusted monitor at deployment, which assumes away part of the Byzantine threat. The authors should state explicitly that this is a 'behavior-detection-based Bayesian defense' whose guarantee does not hold when attackers are indistinguishable. Appendix Table 5 already shows that AUC decreases to 0.5 for λ≥0.8, but the main text does not incorporate this failure mode into its conclusions. This directly determines the strength of the security claim. (2) The experimental comparison and attribution have gaps, and some settings weaken the 'large-scale/worst-case' case. Among the baselines, MTMFQ is the closest, but RoMFAC targets state perturbations, RMFTG is an LQG-robust method, and MIR3 is a small-scale regularizer; none is Byzantine-specific. EIR-MAPPO is omitted for cost, but the paper lacks: (i) an oracle-type upper bound (a policy using the true θ) to separate detection error from control/optimization contributions; (ii) a simple robust-aggregation baseline (e.g., trimmed mean/median or Krum-style aggregation weighted by the detector) to test whether the belief-weighted mean is actually better than classical robust aggregation; and (iii) an adaptive attacker. Moreover, the 'worst-case attacker' in training is actually a VAI-selected subset plus a learned attack policy: the selection α is not optimized, β is sampled uniformly, and the appendix acknowledges that this is scenario coverage rather than a certificate, so the 'worst-case' wording is too strong. On Battle 144/32, RoMFAC (−1402.14) outperforms BeMF (−1450.11), which is slightly at odds with 'consistently'; PowerGrid has only 20/40 agents, which is small for a mean-field method. These gaps affect the assessment of how much of the gain comes from the belief mechanism itself, and how credible the method is in a genuinely adversarial environment.

Questions for authors

Q1. Can you provide an adaptive-attacker experiment in which the attacker knows the detector, imitates cooperative behavior early to obtain a low p_i, and attacks later? When the detector AUC is near 0.5 (Appendix Table 5), does BeMF's gain over w/o-belief disappear? This determines the boundary of the 'Byzantine' claim. Q2. In the proof of Prop. 4.1, the first term contains 1/Z_p; if z_p is much smaller than Z_θ=1−β̄, should the bound be D_S ε(1/z_p+1)/Z_θ? Please correct this or explain why Assumption D.3 allows z_p≥1−β̄. Q3. The detector is supervised with ground-truth type labels; where do these labels come from in deployment, and how sensitive is the method to label noise and attack-family shift? Does the framework retain any guarantee if the external monitor is compromised? Q4. Can the 'worst-case attacker' wording be tightened to 'strong learned attacker / VAI-selected subset'? Is a non-optimized selection α and uniformly sampled β enough to support the abstract's worst-case claim? Q5. On 144 agents/32 attackers, RoMFAC outperforms BeMF; can you explain this? Does the '16/18' count include this configuration? Are 20/40 agents in PowerGrid enough to support 'large-scale'? Q6. Please add an oracle-type upper bound and a robust-aggregation baseline (trimmed mean/median weighted by p_i) to separate detection error from control contributions, and report results under an adaptive attacker. Q7. Does the end-to-end O(N) accounting include the detector's recurrent history for each agent and the central monitor's broadcast cost? Can you report complete wall-clock and communication overhead?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part I)

We thank Reviewer Rgit for the careful reading and constructive feedback. We address each point below and hope our responses clarify the contributions and resolve your remaining concerns.

Q1: Can you provide an adaptive-attacker experiment in which the attacker knows the detector, imitates cooperative behavior early to obtain a low p_i, and attacks later? When the detector AUC is near 0.5 (Appendix Table 6), does BeMF's gain over w/o-belief disappear? This determines the boundary of the 'Byzantine' claim.

Adaptive attacker. Following your suggestion, each Byzantine agent follows the cooperative policy for the first third of the episode to appear benign and obtain a low risk score, then switches to the adversarial policy. The defender and detector remain frozen. Results over two seeds are reported below.

EnvironmentAgent numAdv. numOriginal attacker (Table 1)Adaptive attackerChange
Battle14416−783.23 ± 19.79−755.41 ± 8.55+3.6%
Battle14432−1450.11 ± 21.03−1382.98 ± 10.26+4.6%
Battle14436−1593.89 ± 22.50−1551.57 ± 17.47+2.7%
Battle14448−2162.20 ± 53.74−1801.91 ± 24.98+16.7%
Battle14472−2555.94 ± 26.53−2132.11 ± 6.30+16.6%
PowerGrid40823.72 ± 2.1124.15 ± 0.87+1.8%
PowerGrid401219.75 ± 1.8123.91 ± 1.13+21.1%
PowerGrid401619.21 ± 2.1221.17 ± 0.92+10.2%
PowerGrid402017.38 ± 1.8519.87 ± 1.26+14.3%

The adaptive attacker yields higher mean team returns than the original attacker in all nine configurations: by 2.7–16.7% on Battle and 1.8–21.1% on PowerGrid. We also see belief gradually increases after the attack begins. Rendered rollouts show clear changes in agent behavior after the transition from cooperation to attack. Thus, early imitation does not increase the total damage in these tests.

AUC interpretation. The logit-mixing sweep in Appendix Table 12 (Table 5 in former upload) changes attack behavior as well as detectability. Near-chance AUC therefore does not, by itself, determine whether BeMF's return advantage over w/o belief disappears.

Q2: In the proof of Prop. 4.1, the first term contains 1/Z_p; if z_p is much smaller than Z_θ=1−β̄, should the bound be D_S ε(1/z_p+1)/Z_θ? Please correct this or explain why Assumption D.3 allows z_p≥1−β̄.

Please allow us to clarify the denominator in Proposition 4.1. The apparent dependence on 1/Zp1/Z_p cancels because the corresponding integral is itself proportional to ZpZ_p. We do not require or infer Zp≥1−βˉZ_p \ge 1-\bar\beta from Assumption D.3. With Zp=∫(1−p) dηZ_p=\int(1-p)\,d\eta, Zθ=∫(1−θ) dηZ_\theta=\int(1-\theta)\,d\eta and ε=∫∣p−θ∣ dη\varepsilon=\int|p-\theta|\,d\eta, the integral in the first term is itself proportional to ZpZ_p: since 1−p≥01-p\ge0 and ∣f∣≤DS|f|\le D_S, we have ∣∫(1−p)f dη∣≤DSZp\left|\int(1-p)f\,d\eta\right|\le D_S Z_p. Hence

∣1Zp−1Zθ∣∣∫(1−p)f dη∣≤∣Zp−Zθ∣ZpZθ DSZp=DS∣Zp−Zθ∣Zθ≤DSεZθ.\left|\frac1{Z_p}-\frac1{Z_\theta}\right|\left|\int(1-p)f\,d\eta\right|\le\frac{|Z_p-Z_\theta|}{Z_pZ_\theta}\,D_S Z_p=\frac{D_S|Z_p-Z_\theta|}{Z_\theta}\le \frac{D_S\varepsilon}{Z_\theta}.

The second term is bounded by DSε/ZθD_S\varepsilon/Z_\theta directly, giving W1≤2DSε/Zθ≤2DSε/(1−βˉ)W_1\le 2D_S\varepsilon/Z_\theta\le 2D_S\varepsilon/(1-\bar\beta). Assumption D.3 is used only to ensure Zp>0Z_p>0, so that the weighted field is well defined. We have expanded this step in the proof in Appendix E.3 of the revised paper.

Q3: The detector is supervised with ground-truth type labels; where do these labels come from in deployment, and how sensitive is the method to label noise and attack-family shift? Does the framework retain any guarantee if the external monitor is compromised?

In deployment, the detector sees only observable trajectories, receives no labels, identities or attack fraction at deployment. So we use the setting that type labels are unavailable when deployment. In a power grid, a DER whose inverter firmware or local controller is compromised issues set-points that deviate from the voltage-control objective, which is visible in its telemetry. In a robot swarm, a unit with a faulty actuator or hijacked controller keeps moving but stops following the team's task, which is visible in its trajectory. Such failure modes can be injected in simulation, where each agent's type is known by construction, and this is where the training labels come from.

AuthorsOpenReview ↗

Rebuttal by Authors (Part II)

(reply to Q3)

When facing label noise and attack shift. Our Theorem 4.1 holds for arbitrary selection mechanism α\alpha within the budget β≤βˉ\beta\le\bar\beta, and for every admissible action policy πα\pi_\alpha. So a shift affects BeMF only through the detection error ete_t, which Prop. E.1 bounds by the trace overlap plus the detector's excess risk. Label noise raises the excess risk and weakens the bound smoothly. But our experiments may emperically prove our method. We tested two shifts: random instead of VAI selection and an adaptive attacker (Q1)(full results are in Table 9-11 of our revised paper). The return does not drop under the first two and drops by at most 12.7% under the third.

A trusted detection and aggregation layer is a standard assumption: Byzantine-robust distributed learning assumes a trusted server [1]^{[1]}, and attested cloud platforms assume a trusted verifier [2]^{[2]}. Within our analysis, a monitor that only corrupts the scores pip_i enters the bounds through the realized ete_t, so the guarantee degrades in proportion to the error it introduces. A monitor that tampers with the aggregation itself is outside our threat model, as in the works above.

[1].Yin et al. Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates. ICML 2018.

[2].Schear et al. Bootstrapping and Maintaining Trust in the Cloud. ACSAC 2016. Chen et al. PowerNet: Multi-Agent Deep Reinforcement Learning for Scalable Powergrid Control. IEEE Trans. Power Systems, 2022.

Q4: Can the 'worst-case attacker' wording be tightened to 'strong learned attacker / VAI-selected subset'? Is a non-optimized selection α and uniformly sampled β enough to support the abstract's worst-case claim?

Our attacker is optimized at both levels. VAI optimizes the selection α\alpha, and the action policy πα\pi_\alpha is trained to minimize the defender's return. Sampling β\beta uniformly is used only in training, to cover attack budgets; each reported result is evaluated at a fixed β\beta.

What the experiments instantiate is VAI's attacker. It is a computed, optimized attacker, not a certified global worst case, so we now write VAI-selected compromised sets and a "learned adversarial action policy". VAI is a strong available instantiation for three reasons. Its decomposition of the selection problem provably preserves the optimal solution of the bilevel attack [1]^{[1]}. It outperforms degree-centrality, evolutionary-search, graph-embedding and other selectors on 17 of 18 tasks in its own evaluation. In our tests, replacing VAI selection with random selection raises the defender's return in four of five settings. The attacker's policy is also optimized against the deployed defense. Every reported result freezes the final defender and trains the attacker against it, which lowers the return by a further 3.6–7.8% relative to the attacker at the end of alternating training(Table 11). On the theory side, Corollary F.2.1 already keeps the optimization residual εopt\varepsilon_{\mathrm{opt}} instead of assuming exact minimax training.

We revised the abstract Section 5 and Appendix G.1 accordingly, and added the random-selection, intimated attacker (Q1) results to Appendix G.

[1].Li et al. Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning. arXiv:2509.15103, 2025.

Q5: On 144 agents/32 attackers, RoMFAC outperforms BeMF; can you explain this? Does the '16/18' count include this configuration? Are 20/40 agents in PowerGrid enough to support 'large-scale'?

Thank you for highlighting this exception. Yes, the Battle configuration with N=144N=144 and K=32K=32 is included in the 18 configurations. In this setting, RoMFAC achieves a higher mean return than BeMF (-1402.14 versus -1450.11). The other exception is PowerGrid with N=20N=20 and K=8K=8. Thus, “16 out of 18” already accounts for both exceptions and refers to the highest reported mean return.

We do not claim that BeMF outperforms every baseline in every configuration. In the same setting, BeMF with the true types in place of the detector (the oracle in our response to Q6) reaches -1385.35 ± 20.34. BeMF's average loss to detection error there is larger than other cases, so the shortfall can explained by imperfect detection, not by the belief-weighted control.

AuthorsOpenReview ↗

Rebuttal by Authors (Part III)

(Reply to Q5)

Regarding the scale of PowerGrid, we take “large-scale” to be relative to the environment. What matters is how hard the coupled system is to simulate and control at that size, not the agent count alone. In PowerGrid, each agent controls a distributed generator in a physically coupled electrical network. Each environment step numerically integrates hundrreds of nonlinear dynamics of generator states, line currents and load currents, an ODE system of dimension 281 at N=20N=20 and 565 at N=40N=40. Agents regulate voltage under load disturbances from local observations, while each action affects the other units through the network. By the standard of this domain, 20 and 40 generators are large: PowerNet’s base case [1]^{[1]} has 6 generators and it describes its 40-generator setting as large scale (§V-D), Ma et al.[2]^{[2]} evaluate the same 20/40 settings, and MAPDN [3]^{[3]} describes its 38-agent setting as large scale.

[1].Chen et al. PowerNet: Multi-Agent Deep Reinforcement Learning for Scalable Powergrid Control. IEEE Trans. Power Systems, 2022.

[2].Ma et al. Efficient and Scalable Reinforcement Learning for Large-Scale Network Control. Nature Machine Intelligence, 2024.

[3].Wang et al. Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks. NeurIPS 2021.

Q6 Please add an oracle-type upper bound and a robust-aggregation baseline (trimmed mean/median weighted by pip_i) to separate detection error from control contributions, and report results under an adaptive attacker.

We added both baselines on Battle with 144 agents (Tables 2 and 8). Oracle belief replaces the scores with the true types, pi=θip_i=\theta_i. Trimmed mean uses the same trained detector, removes the kk agents with the highest pip_i, and averages the rest.

Adv Numw/o BeliefNo-mean fieldNo-weight-mean fieldTrimmed meanBeMFOracle belief
16-2450.65 ± 122.52-1337.68 ± 23.10-1081.99 ± 18.77-936.76 ± 19.33-783.23 ± 19.79-732.99 ± 11.05
32-2739.41 ± 122.58-1786.58 ± 34.22-1568.15 ± 29.12-1431.53 ± 12.55-1450.11 ± 21.03-1385.35 ± 20.34
36-2809.25 ± 74.69-1871.95 ± 33.70-1787.78 ± 41.05-1819.64 ± 9.55-1593.89 ± 22.50-1517.77 ± 24.40
48-2569.90 ± 116.71-2576.57 ± 68.91-2321.13 ± 51.80-2295.54 ± 14.57-2162.20 ± 53.74-1928.85 ± 26.11
72-2905.39 ± 101.47-2739.61 ± 42.15-2659.62 ± 38.76-2624.78 ± 16.52-2555.94 ± 26.53-2327.57 ± 26.84

Detection error (BeMF vs. oracle). The gap to the oracle is the cost of detection error, which Theorem 4.1 bounds by KT∑tγtetK_T\sum_t\gamma^te_t. With up to 36 adversaries it is 50–76, and BeMF recovers 94–97% of the oracle's gain over w/o belief (Table 7). With 48 and 72 adversaries the gap grows to 228–233 and the recovery falls to 60–64%. Prop. 4.1 predicts this: the field error scales with et/(1−βˉ)e_t/(1-\bar\beta), so the same detection error costs more as β\beta grows, and ete_t itself rises because the population the detector compares each agent against is increasingly corrupted.

Soft weighting vs. trimming (BeMF vs. trimmed mean). BeMF is higher by 69–226 in four of five settings. A misranked agent costs trimming a full unit of weight, an adversary kept or a cooperative agent removed, but costs soft weighting only ∣pi−θi∣|p_i-\theta_i|. With 32 adversaries the trimmed mean is 19 higher, within one standard deviation. Trimming can match soft weighting when its top-kk ranking is accurate, but it needs the adversary count kk, which BeMF does not.

Control. The oracle's own return falls from −733 to −2328 as adversaries increase. The oracle knows every type, so this loss is a limit of control, not of detection.

Adaptive attacker. See Q1.

Reviewer_3t85OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes BeMF, a framework for Byzantine-robust large-scale multi-agent reinforcement learning. It combines neural type inference, belief-weighted mean-field aggregation, and adversarial PPO. Experiments on Battle and PowerGrid show promising results, but the methodological novelty, robustness against adaptive attacks, and practical theoretical guarantees remain insufficiently established.

Critical strengths

1.The paper addresses robustness in large-scale multi-agent systems with unknown adversarial agents. 2.Belief-weighted aggregation provides an intuitive approach to reducing the influence of potentially compromised agents.

Critical weaknesses

1.The paper does not sufficiently demonstrate the necessity of combining type inference, belief-weighted aggregation, and adversarial PPO over simpler alternatives. The theoretical analysis mainly establishes conditional performance-stability guarantees rather than convergence of the practical algorithm to a robust equilibrium. 2.The experiments primarily rely on a VAI-derived attack-selection mechanism, leaving generalization to unseen or adaptive attacks insufficiently explored. It remains unclear whether the detector can identify attackers that mimic cooperative behavior or evade detection. 3.The paper lacks a direct comparison with EIR-MAPPO at the same population scale. Existing ablations also fail to disentangle the individual contributions of type inference, belief-weighted aggregation, and adversarial training, making it difficult to identify the sources of performance gains and establish the necessity of the proposed framework. 4.The figures, particularly the method overview, are visually cluttered and lack a clear visual hierarchy.

Questions for authors

1.What are the key advantages of BeMF over simpler alternatives, and does the practical algorithm have convergence guarantees? 2.Can BeMF defend against unseen or adaptive attacks that evade detection? 3.Can the authors provide a same-scale comparison with EIR-MAPPO and more detailed component-wise ablations?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Q1, Q2)

We thank Reviewer 3t85 for the review. We clarify the theoretical contribution and report additional experiments below. Also we modify our figure and its description to make it easier to understand. Revisions are marked in blue in the revised PDF.

Q1: What are the key advantages of BeMF over simpler alternatives, and does the practical algorithm have convergence guarantees?

BeMF has three advantages over simpler alternatives: higher return under attack, cost linear in NN, and a performance bound under imperfect detection.

  • Higher return under attack. A standard mean-field policy takes a raw field that mixes cooperative and Byzantine behavior, and adversarial training alone does not remove the Byzantine part of this input. BeMF weights each agent’s contribution by 1−pti1-p_t^i. With the same adversarial training, it outperforms w/o belief in all 11 Battle configurations (Table 7); at 144 agents and 16 attackers, return improves from −2450.65 to −783.23. Simpler uses of the same detector are also worse: giving the policy the risk score with the raw field is 104–299 lower (Q3), and hard-removing the kk highest-risk agents is 69–226 lower in four of five settings and needs the attacker count kk, which BeMF does not.
  • Linear cost. Exact joint type beliefs range over 2N2^N configurations, and observer–target beliefs need O(N2)O(N^2) entries. BeMF keeps one risk score per agent, so inference is O(N)O(N) and the policy interface does not depend on NN. At 16 agents it runs at 1809 environment steps/s, against 59.5 for EIR-MAPPO (Table 6).
  • A bound under imperfect detection. Proposition 4.1 bounds the distortion of the weighted field by the population-average type-score error ete_t, and Theorem 4.1 bounds the return gap to the same controller supplied with true types by KT∑tγtetK_T\sum_t\gamma^t e_t. The raw field has no such bound, because the attacker controls its distortion.

In the revised paper, we add Theorem F.3, a local convergence guarantee for idealized alternating updates with a frozen detector. Near a nondegenerate local solution of the robust sup–inf objective, and with enough adversary steps per defender step, the iterates converge linearly, up to a residual proportional to the gradient error. The resulting exploitability bound carries over to other population sizes with an added finite-population error, O(N−1/2)O(N^{-1/2}) for finite state spaces.

Q2: Can BeMF defend against unseen or adaptive attacks that evade detection?

Under the assumptions of alternating updates with a frozen detector, Theorem 4.1 applies to admissible selection mechanisms and attacker policies, with ete_t evaluated under the actual attack. Proposition E.1 gives et≤At+εte_t\le A_t+\varepsilon_t, separating behavioral overlap from detector excess risk. Transfer therefore depends on detection quality under the new attack. In our action-only threat model, camouflage through cooperative actions can also reduce damage. We test both aspects with the defender and detector frozen.

We train with VAI because it targets vulnerable subsets and outperforms random and other selectors on 17 of 18 tasks in Li et al. (2025).

Adv. numVAI selection (Table 1)Random selectionIntimate attacker
16−783.23 ± 19.79−1006.72 ± 253.11−755.41 ± 8.55
32−1450.11 ± 21.03−1264.43 ± 241.02−1382.98 ± 10.26
36−1593.89 ± 22.50−1553.81 ± 192.37−1551.57 ± 17.47
48−2162.20 ± 53.74−1742.35 ± 158.90−1801.91 ± 24.98
72−2555.94 ± 26.53−2360.97 ± 167.56−2132.11 ± 6.30
  • Unseen selection: replacing VAI with random selection on Battle (N=144N=144) yields higher mean returns in four of five configurations. At 16 attackers, return is −1006.72 ± 253.11 versus −783.23 ± 19.79 under VAI: the mean drop is 223.49, compared with a standard deviation of 253.11 under random selection.
  • Intimate attacker: Byzantine agents cooperate for the first third of an episode, then attack. Across two seeds, return is higher in all nine configurations: by 2.7–16.7% on Battle (N=144N=144) and 1.8–21.1% on PowerGrid (N=40N=40). Early imitation does not make the subsequent attack more damaging in these tests.
  • Camouflaged attacker: mixing toward cooperative behavior brings detection to chance at λ≥0.8\lambda\ge0.8 (Table 12). Inspected rollouts show that these agents also stop attacking teammates. This supports a camouflage–damage trade-off for the tested attack.
AuthorsOpenReview ↗

Rebuttal by Authors (Q3)

Q3. Can the authors provide a same-scale comparison with EIR-MAPPO and more detailed component-wise ablations?

In fact, these two methods are hard to compare since they differ in backbone algorithm, evaluation environment and running time. (1) Backbone: EIR-MAPPO builds on MAPPO with identity-indexed beliefs, while BeMF is a mean-field actor–critic, so a direct return comparison would mix backbone and defense differences. (2) Environment: EIR-MAPPO is evaluated with at most four agents and one adversary, versus our 20–144 agents and up to 50% corruption, so it would have to be adapted to our environments. (3) Running time: even at 16 controlled agents, EIR-MAPPO reaches 59.5 environment steps/s versus 1809 for BeMF, i.e., 7.8 versus 0.26 GPU-days per 40M-step run, and it needs 14.6 and 29.2 GPU-days at 32 and 64 agents (Table 6; Appendix G.1, "Methods we could not run"). A full comparison is therefore infeasible within the discussion period.

To address your concerns, we have included an experiment which approximates EIR-MAPPO by removing mean-field aggregation from BeMF while keeping its type inference (w/o mean field): the policy still receives the Byzantine risk score but no population field. This keeps the belief-based defense that EIR-MAPPO relies on, while holding the backbone, environment and adversarial training fixed. On Battle with 144 agents (three seeds), BeMF outperforms this variant in all five configurations, by 184–554 in return, so mean-field aggregation adds to belief-based defense at our scale.

For the component-wise ablations, we add w/o weighted field, which keeps the risk score and the raw field. BeMF > w/o weighted field > w/o mean field holds in all five configurations: belief weighting improves return by 104–299, and keeping the raw field over removing all population information improves it by 80–256. Removing all type information (w/o belief) is worst in four of five settings. Both inference and belief-aware aggregation therefore contribute beyond adversarial training alone.

AuthorsOpenReview ↗

Invitation to Author- Reviewer Discussion

Dear Reviewer

We are the authors of Submission 5649: "Robust Large-Scale Multi-agent Reinforcement Learning against Byzantine Fault". We have posted a point-by-point response to your comments, which we believe addresses your concerns. For your convenience, this comment adds the per-configuration numbers behind our answer to Q3, which we had summarized there as ranges. All of these results are already in the revised appendix (marked in blue), and this comment makes no new claims.

Table A. Component ablations on Battle (N = 144 controlled agents, K compromised). Team return, mean ± std, higher is better. The two new variants (w/o mean field, w/o weighted field) use three seeds.

Kw/o beliefw/o mean fieldw/o weighted fieldw/o population contextBeMF
16−2450.65 ± 122.52−1337.68 ± 23.10−1081.99 ± 18.77−947.46 ± 8.38−783.23 ± 19.79
32−2739.41 ± 122.58−1786.58 ± 34.22−1568.15 ± 29.12−1530.59 ± 19.74−1450.11 ± 21.03
36−2809.25 ± 74.69−1871.95 ± 33.70−1787.78 ± 41.05−1820.85 ± 11.26−1593.89 ± 22.50
48−2569.90 ± 116.71−2576.57 ± 68.91−2321.13 ± 51.80−2376.76 ± 19.67−2162.20 ± 53.74
72−2905.39 ± 101.47−2739.61 ± 42.15−2659.62 ± 38.76−2624.37 ± 20.68−2555.94 ± 26.53

In all five configurations, BeMF > w/o weighted field > w/o mean field. Each gap in the first two columns is larger than the sum of the two standard deviations involved. w/o belief is the worst variant in four of the five settings. The exception is K = 48, where it is within 6.7 return units of w/o mean field, which is within one standard deviation. w/o population context replaces the detector’s pooled context with a constant. Removing it lowers return by 68–227, so the population context used for inference also contributes.

Table C. Cost of one training run (single A800-80GB, 40M environment steps; Table 6, Appendix G.1).

NBeMF steps/sEIR-MAPPO steps/sBeMF GPU-daysEIR-MAPPO GPU-daysRatio
16180959.50.267.830×
32143331.70.3214.645×
64117015.90.4029.274×
1447367.1*0.6365.4*—

As the author–reviewer discussion phase is ending soon, we would greatly appreciate it if you could share your thoughts on our rebuttal. We would be happy to clarify any remaining points.

Best regards,

Authors of Submission 5649

AuthorsOpenReview ↗

Follow-up: Discussion Phase Ending Soon

Dear Reviewer 3t85,

Thank you again for your time and for your detailed review. We understand this is a busy period.

The author–reviewer discussion phase ends in less than 10 hours, so we wanted to follow up once more. Have our responses, including the additional Q3 numbers we posted earlier, addressed your concerns? If anything is still unclear, please let us know. We will respond promptly within the remaining time.

If our responses have resolved your concerns, we would be grateful if you could consider taking them into account in your final assessment.

Best regards,

Authors of Submission 5649

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.