acceptodds
Under review as a conference paper at ICLR 2027

-Sparse Wasserstein Distributionally Robust Value Factorization for Cooperative MARL

Abstract

Robust value factorization for MARL under concurrent actuator faults is combinatorial: allowing up to failures among agents creates fault patterns per Bellman backup. We introduce -Sparse Wasserstein Distributionally Robust MARL (KW-DRO), which limits each fault realization to agents and uses a Wasserstein radius to control the expected fault count. For the standard single-hidden-layer QMIX mixer, we show that learner-side fault damage is monotone submodular under convex activation and nonnegative utility drops. KW-DRO therefore constructs candidates in time with a per-cardinality guarantee, exactly for VDN, and solves probability allocation by an exact lower-convex-hull computation. The solver agrees with reference linear programs, and greedy search matches exhaustive selection on all tested QMIX states. On RWARE, nominal VDN exhibits late policy collapse in long training runs; KW-DRO at keeps all runs alive, compared with under the nominal backup. Matched controls attribute this stabilization to moderate pessimism rather than Wasserstein geometry alone. Anonymous code available: https://anonymous.4open.science/r/kw-dro.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, robust optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oMRcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows the effectiveness of KW-DRO in saving the computational complexity of tolerating agent failures in MARL.

Critical strengths

The introduced sub modularity idea sounds somewhat interesting, while the results shows its effectiveness over nominal solutions.

Critical weaknesses
  1. The authors mainly test the advantage of KW-DRO over nominal, full backup and show the computational savings and performance improvement. But I am afraid that some simpler alternatives, including DrIGM, shall be incorporated. More alternatives can be adding some randomness to the training process.

  2. The description of the paper is somewhat misleading, as the Wasserstein distance can be simply regarded as a budget. So, the authors shall clarify the implications of adopting Wasserstein distance here.

Minor: The paper is not so easy to follow, with inconsistent paragraph statements. More polishing is required.

Questions for authors
  1. The authors mainly test the advantage of KW-DRO over nominal, full backup and show the computational savings and performance improvement. But I am afraid that some simpler alternatives, including DrIGM, shall be incorporated. More alternatives can be adding some randomness to the training process.

  2. The description of the paper is somewhat misleading, as the Wasserstein distance can be simply regarded as a budge. So, the authors shall clarify the implications of adopting Wasserstein distance here.

Minor: The paper is not so easy to follow, with inconsistent paragraph statements. More polishing is required.

4 comments
AuthorsOpenReview ↗

Response to Reviewer oMRc

Thank you for your review and for noting the appeal of our submodular formulation and its gains over nominal training.

We have uploaded a revised PDF for rebuttal, and highlighted the edits in blue.

Q1. Simpler alternatives: DrIGM, Double Q, and adding randomness to training. We agree that KW-DRO should be tested against simpler robust and randomized baselines. The revision evaluates all three across VDN (Sec. 5.2, Table 1; App. C.6) and a new 16-seed-per-arm QMIX study (162 runs at 10710^7 steps; Sec. 5.3, App. I):

  1. DrIGM (Qu et al., 2026) and Double Q on VDN (Table 1, 10 seeds/layout): Under DrIGM's vanishing-minimal-value condition, its ρ\rho-contamination target (Eq. 8 of its arXiv version) is r+γ(1−ρ)E[max⁡a′Qtot(s′,a′)]r+\gamma(1-\rho)\mathbb{E}[\max_{a'}Q_{\mathrm{tot}}(s',a')], which for VDN equals our scalar-shrinkage backup r+(1−c)γVtot(s′)r+(1-c)\gamma V_{\mathrm{tot}}(s') at ρ=c\rho=c. Table 1 marks these rows (DrIGM-contamination, ρ∈{0.0036,0.0072}\rho\in\{0.0036,0.0072\}). At ρ=0.5\rho=0.5, KW-DRO keeps 20/20 runs alive vs. 14/20 nominal (p=0.020p=0.020) and 18/20 DrIGM-contamination (p=0.50p=0.50), with Double Q also at 20/20. At ρ=1.0\rho=1.0, KW-DRO's state-dependent penalty stalls early exploration (9/20 vs. 20/20 for uniform shrinkage): moderate pessimism stabilizes VDN, while state-dependent targets need a calibrated radius (Fig. 4).

  2. Adding randomness during training (Fault-Randomized Training) — VDN and new 16-seed QMIX study: In App. C.6 (Table 3), VDN trained with random stalls and the nominal backup kept 3/5 runs alive on small-8ag (7/10 pooled vs. 10/10 for KW-DRO, p=0.211p=0.211). To test this with higher power on nonlinear mixing, we ran a 16-seed-per-arm QMIX study at 10710^7 steps (Sec. 5.3, App. I, Tables 13–15) comparing nominal QMIX, transient KW-DRO (ρ=0.5\rho=0.5), persistent KW-DRO (D=10D=10, dose-matched ρ=0.05\rho=0.05), and fault-randomized training (random multi-step stalls during rollouts with nominal QMIX):

    • On tiny-4ag (K=2K=2), fault-randomized training barely improves over nominal QMIX (24.0 vs. 22.4 deliveries/1k steps fault-free, 4/16 vs. 3/16 competent seeds; 21.0 vs. 19.7 under targeted 20-step KK-robot outages). Both transient KW-DRO (45.3 clean, 35.7 under outages, 11/16 competent) and persistent KW-DRO (48.9 clean, 38.0 under outages, 11/16 competent) double throughput, and persistent KW-DRO beats fault-randomized training by +17.0+17.0 deliveries/1k steps under outages (p=0.005p=0.005).
    • A plausible reason: RWARE rewards are very sparse (0.003%0.003\% of random agent-transitions, App. C.1), so stalling robots during rollouts can break coordinated deliveries and thin out rewarding trajectories in replay, whereas KW-DRO keeps rollouts clean and applies worst-case stalls only inside the backup. Consistently, FR learns as slowly as nominal QMIX (Fig. 13).
    • On small-8ag (K=3K=3), persistent KW-DRO also leads nominal QMIX and fault-randomized training in mean throughput under outages (34.9 vs. 24.6 and 28.0), though not resolved at 16 seeds.

Q2. Viewing the Wasserstein distance as a budget, and its implications. We agree: around P^=δ0\hat{P}=\delta_0 under the Hamming metric, Lemma 5 gives W1(P,δ0)=EP[∥ξ∥1]W_1(P,\delta_0)=\mathbb{E}_P[\|\xi\|_1], so ρ\rho budgets expected stalled actuators per step. We revised Sec. 1 and added a paragraph after Definition 1 stating this up front. The support-constrained W1W_1 set has four implications:

  1. Why W1W_1 around δ0\delta_0: KL/χ2\chi^2 balls collapse to {δ0}\{\delta_0\}, and TV/contamination bound only P(ξ≠0)P(\xi\neq 0), charging KK simultaneous stalls the same as one. Hamming-W1W_1 prices faults per stalled actuator.
  2. Need for both ρ\rho and KK: EP[∥ξ∥1]≤ρ\mathbb{E}_P[\|\xi\|_1]\le\rho alone still allows mass ρ/N\rho/N on NN simultaneous stalls. Adding supp(P)⊆ΞK\text{supp}(P)\subseteq\Xi_K enforces sparsity while keeping allocation an exact 2D lower-hull LP on at most two atoms (Theorem 7).
  3. Interaction across durations: Under 1-step stalls, decreasing submodular marginal damages make the greedy curve convex, so at ρ≤1\rho\le 1 the hull uses only k∈{0,1}k\in\{0,1\} (min⁡(K,ρ)\min(K,\rho), Prop. 3). Under DD-step outages (Sec. 3.4, App. H), ρ\rho budgets new onsets while ongoing stalls accumulate against KK, activating KK even at ρ≤1\rho\le 1 (Example 16, Fig. 15d).
  4. Unequal costs and beyond a Dirac nominal: With unequal per-actuator costs around δ0\delta_0, W1W_1 remains a linear (weighted) budget and our hull solver stays exact (Alg. 2; Sec. 5.1, App. C.10). For a non-Dirac empirical nominal (e.g., fitted from hardware logs), W1W_1 is no longer a linear budget (counterexample in App. A.1); estimating that geometry from operational fault data is our next step (Sec. 6).

Minor: Presentation. We polished Sec. 1, Sec. 3.1, and Sec. 5 for consistent terminology and added a Notation Table in Appendix G (Table 12). Again, new edits are highlighted blue in revised PDF.

Thanks for your input. In my opinion, if the Wasserstein distance can be simplified as a budget, the authors are suggested compare it with this simple idea and show its advantages. Also the readability issue remains.

Thank you for following up with us. Your comment highlights an important misunderstanding that our shorthand word "budget" in the previous message. We clarify why Wasserstein DRO differs from a simple discrete budget, where W1W_1 sits in the complexity spectrum, and how the two compare directly in our experiments.

1. Wasserstein distance (over distributions) vs. a simple budget (like Hamming distance on vectors). A discrete fault budget and a Wasserstein ball act on two different mathematical spaces:

  • Simple discrete budget (Hamming ball ∥ξ∥1≤B\|\xi\|_1 \le B on {0,1}N\lbrace 0,1\rbrace^N): This is a deterministic worst-case constraint on individual fault vectors, min⁡∥ξ∥1≤BVξ(s′)\min_{\|\xi\|_1 \le B} V^{\xi}(s'). Because B∈{1,2,…,N}B \in \lbrace 1,2,\dots,N\rbrace is an integer, even the smallest non-zero budget (B=1B=1) forces a worst-case robot stall on 100% of Bellman backups, and cannot represent rare or fractional hardware fault rates.
  • Wasserstein DRO (W1(P,P^)≤ρW_1(P,\hat{P}) \le \rho on Δ(ΞK)\Delta(\Xi_K)): Here, Hamming distance is only the ground metric between realizations, while W1(P,P^)=inf⁡Π∈Γ(P,P^)E[∥ξ−ξ′∥1]W_1(P,\hat{P}) = \inf_{\Pi\in\Gamma(P,\hat{P})}\mathbb{E}[\|\xi-\xi'\|_1] measures transport cost between probability distributions. KW-DRO optimizes continuous adversarial weights p∈ΔM−1p \in \Delta^{M-1} across fault patterns via a 2D lower convex hull (Theorem 7). Around P^=δ0\hat{P}=\delta_0 (∑mpmcm≤ρ\sum_m p_m c_m \le \rho), W1W_1 optimizes a distribution mixing fault-free and faulted outcomes to match realistic fault rates (e.g., ρ=0.5\rho=0.5 or ρ=0.05\rho=0.05) while KK caps concurrent stalls; for any empirical nominal P^≠δ0\hat{P}\neq\delta_0, W1(P,P^)≤ρW_1(P,\hat{P}) \le \rho requires full optimal-transport coupling and does not reduce to any scalar budget (counterexample in App. A.1).
  • Why W1W_1 is already the minimal DRO setup: Within distributional transport, first-order W1W_1 is the simplest linear formulation, keeping the inner backup convex. Richer relational transport metrics — such as Gromov–Wasserstein distance across agent interaction structures — turn the coupling problem into a non-convex quadratic program, which would be prohibitive to solve at every Bellman backup. W1W_1 is thus the simplest distributional model that avoids deterministic over-conservatism while preserving an exact, fast inner solver.
    • A nice, but also a bit long to read and study about DRO:
      • Kuhn D, Shafiee S, Wiesemann W. Distributionally robust optimization. Acta Numerica. 2025;34:579-804. doi:10.1017/S0962492924000084

2. Direct empirical comparison against "simple budget" baselines. Because the candidate curve k↦vkk \mapsto v_k is decreasing and convex for VDN, the simple deterministic Hamming-budget backup min⁡∥ξ∥1≤BVξ(s′)\min_{\|\xi\|_1 \le B} V^{\xi}(s') (stalling up to BB worst-case robots without distributional weighting) is algebraically identical to setting integer ρ=B\rho = B (B≤K=3B \le K=3) in Table 1 and Figure 4:

MethodFormulationVDN Survival (Table 1 / Fig. 4)Fault Deployment Throughput (Sec. 5.2)
KW-DRO (ρ=0.5,K=3\rho=0.5, K=3)Distributional W1W_1 over Δ(Ξ3)\Delta(\Xi_3)20/20 (Table 1) / 15/15 (Fig. 4)9.2 deliveries / 1k steps
Simple Hamming budget (B=1B=1)Deterministic min⁡∥ξ∥1≤1Vξ(s′)\min_{\|\xi\|_1 \le 1} V^{\xi}(s') (ρ=1\rho=1)9/20 (11/20 fail to learn)3.9 deliveries / 1k steps
Simple Hamming budget (B=2,3B=2, 3)Deterministic min⁡∥ξ∥1≤BVξ(s′)\min_{\|\xi\|_1 \le B} V^{\xi}(s') (ρ=2,3\rho=2, 3)0/15 (Fig. 4; none learn)—

Without distributional W1W_1 weighting, even a B=1B=1 discrete budget assumes a robot fails at every step, making targets overly pessimistic and preventing learning on most seeds (9/20 at B=1B=1, 0/15 at B≥2B\ge 2), whereas distributional KW-DRO keeps all 20/20 runs alive (14/20 nominal, p=0.020p=0.020). The same holds on QMIX under persistent D=10D=10 outages (App. I): W1W_1 lets us pair a small per-step onset probability (ρ=0.05\rho=0.05) with concurrency cap K=2K=2, doubling throughput over nominal QMIX (38.0 vs. 19.7 under targeted outages, p=0.010p=0.010) and beating a fixed rollout fault-rate budget (fault-randomized training, 21.0, p=0.005p=0.005). Even on the simplex itself, replacing our exact 2D lower-hull W1W_1 solver with a simple greedy budget heuristic (ratio-greedy, sorting candidates by damage-per-cost) is suboptimal under unequal actuator costs on our 2,000-LP benchmark (Sec. 5.1, App. C.10).

3. Readability. We appreciate your points to feedback on readability. During the rebuttal window, we prioritized spelling out every assumption, derivation, and control experiment in full detail so all reviewers could verify the technical claims rigorously, which made the revision denser rather than shorter. For the final version, we will streamline the main text — moving secondary technical caveats and diagnostic details to the appendix and tightening Sections 3–5 so the core intuition and method read cleanly from start to finish.

Thanks for the comment and the efforts. I will maintain my score, due to the remaining concern on readability.

Reviewer_9psGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the problem of MARL with simultaneous actuator faults. The authors model the failure distribution using a sparse Wasserstein ambiguity set and exploit the submodularity of the fault damage under the value factorization framework. The authors then provides a probability allocation algorithm based on lower convex hulls to construct robust bellman backup. Finally, numerical results are provided to validate the theoretical findings.

Critical strengths
  • The problem setting studied in this paper is practically relevant and well-motivated.
  • Extensive numerical results are provided to validate the theoretical findings.
Critical weaknesses
  • One of my major concerns with this manuscript is its lack of clarity. The paper is difficult to follow, with several concepts and notations introduced without clear definition. The current presentation makes it challenging to fully evaluate the technical novelty and significance of the proposed contributions. I believe the the current form of this manuscript falls below the presentation standards expected at ICLR.
  • Although MARL with concurrent actuator failures is a well-motivated application, the scope of the paper remains relatively narrow. In particular, the authors do not apply their framework to broader cooperative MARL settings. Furthermore, the choice of the sparse Wasserstein ambiguity set is insufficiently justified, particularly regarding the assumptions with the Wasserstein radius.
  • Some important related works are not discussed in this paper. For example, [1] already studies robust value factorization in MARL settings. While the model considered in this paper is different, I believe it is necessary for the authors to highlight the similarities and distinctions from prior existing works.

[1] Qu, Chengrui, et al. "Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorization." arXiv 2026.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9psG

Thank you for your constructive review and for noting both the practical relevance of concurrent actuator failures and our numerical validation. Below we address DrIGM, notation and clarity, ambiguity-set assumptions, and scope.

We have uploaded a revised PDF for rebuttal, and highlighted the edits in blue.

1. Discussion and comparison with DrIGM (Qu et al., 2026). Thank you for highlighting DrIGM. We cited and discussed the work in our original Sec. 2 and App. D (Table 11), but under its ICLR 2026 proceedings title ("... with Value Factorization", Qu et al., 2026) rather than its arXiv title ("... via Robust Value Factorization"), making it easy to miss in the bibliography. Prompted by your comment, we expanded both our conceptual and empirical comparison with DrIGM (Sec. 2, Sec. 5.2, App. D):

  • Where uncertainty enters: DrIGM places (h,a)(h,a)-rectangular TV or ρ\rho-contamination sets on the environment transition kernel. KW-DRO keeps the nominal kernel P0P_0 and places a support-constrained Wasserstein set on sparse post-transition actuator stalls (ξ∈ΞK\xi\in\Xi_K, forcing at most KK robots to noop).
  • Role of factorization: DrIGM designs robust per-agent utilities and mixing to preserve robust IGM consistency. KW-DRO uses standard VDN or convex monotone QMIX utilities and proves that fault damage g(S)=f(q)−f(qS)g(S)=f(q)-f(q^S) is monotone submodular (Lemma 1), reducing the ∑k≤K(Nk)\sum_{k\le K}\binom{N}{k} search to O(NK)O(NK) greedy calls with a (1−1/e)(1-1/e) per-cardinality bound (exact for VDN) plus an exact 2D lower-hull LP solver (Theorem 7).
  • Direct comparison in Table 1: Under DrIGM's vanishing-minimal-value condition, its ρ\rho-contamination target (Eq. 8 of arXiv:2602.11437) is r+γ(1−ρ)E[max⁡a′Qtot(s′,a′)]r+\gamma(1-\rho)\mathbb{E}[\max_{a'}Q_{\mathrm{tot}}(s',a')], which for VDN equals our scalar-shrinkage backup r+(1−c)γVtot(s′)r+(1-c)\gamma V_{\mathrm{tot}}(s') at ρ=c\rho=c. Table 1 marks these rows (DrIGM-contamination, ρ∈{0.0036,0.0072}\rho\in\{0.0036,0.0072\}, 10 seeds/layout), alongside Double Q; Table 11 also contrasts action-robust and resilient MARL.

2. Clarity and notation. We revised Sec. 1 and Sec. 3.1 so every symbol is defined before first use, and added a complete Notation Table in Appendix G (Table 12). KW-DRO modifies only the TD bootstrap target on replay tuples (s,a,r,s′)(s,a,r,s'), leaving network architecture and decentralized greedy execution unchanged:

  1. At s′s', evaluate each robot's healthy greedy utility qi=max⁡aiQi(τi′,ai)q_i=\max_{a_i}Q_i(\tau'_i,a_i) and stalled utility q~i=Qi(τi′,noop)\tilde{q}_i=Q_i(\tau'_i,\mathtt{noop}) (giving drops qi−q~i≥0q_i-\tilde{q}_i\ge 0).
  2. Build a nested fault chain ∅=S0⊂S1⊂⋯⊂SK\emptyset=S_0\subset S_1\subset\dots\subset S_K by greedily adding the robot whose stall most reduces fmix(qS;s′)f_{\mathrm{mix}}(q^S;s') (KK batched rounds, O(NK)O(NK) mixer calls; Alg. 1).
  3. Compute the worst-case expectation over the K+1K+1 candidate values at radius ρ\rho via a 2D lower convex hull (Alg. 2) to form yDROy_{\mathrm{DRO}}.

3. Sparse Wasserstein ambiguity set and assumptions on radius ρ\rho. The sole assumption behind Lemma 5 (W1(P,P^)=EP[∥ξ∥1]W_1(P,\hat{P})=\mathbb{E}_P[\|\xi\|_1]) is that the nominal fault channel is the fault-free point mass P^=δ0\hat{P}=\delta_0 under the Hamming metric ∥ξ∥1\|\xi\|_1. Around δ0\delta_0, KL/χ2\chi^2 balls collapse to {δ0}\{\delta_0\}, while TV/ρ\rho-contamination bound only P(ξ≠0)P(\xi\neq 0), charging KK simultaneous stalls the same as one. Hamming-W1W_1 prices faults per stalled actuator so ρ\rho directly measures expected stalls per step, while supp(P)⊆ΞK\text{supp}(P)\subseteq\Xi_K caps simultaneous stalls per realization at KK. Prop. 3 and Fig. 4 characterize the radius window: under 1-step stalls, concurrent atoms (k>1k>1) activate for ρ>1\rho>1, whereas in our new DD-step persistent-outage extension (Sec. 3.4, App. H), ongoing stalls accumulate so KK shapes the value function even at ρ≤1\rho\le 1, governed by expected dose ρD\rho D (Prop. 15(iv)).

4. Scope and new 16-seed QMIX study (162 runs at 10710^7 steps). Lemma 1, Corollary 2, and Theorem 7 depend only on the mixer (VDN or convex monotone QMIX) and a noop/stop action, applying directly to cooperative MARL tasks with those elements (e.g., LBF, MPE, SMAC). While 10710^7-step runs on external suites exceeded the discussion window, we expanded our empirical study with a 16-seed-per-arm QMIX evaluation under persistent multi-step outages (162 runs at 10710^7 steps, protocol fixed before any final checkpoint; Sec. 5.3, App. I). On tiny-4ag, transient and persistent KW-DRO more than double nominal QMIX throughput (45.3 and 48.9 vs. 22.4 deliveries/1k steps fault-free, p≤0.019p\le 0.019; 35.7 and 38.0 vs. 19.7 under targeted 20-step outages, p≤0.030p\le 0.030), and persistent KW-DRO beats fault-randomized training (38.0 vs. 21.0, p=0.005p=0.005). On the trained networks, greedy matches exhaustive search within 1.4×10−51.4\times 10^{-5} on 2,904 DD-step cases (Fig. 15b).

I thank the authors for their response. However, I remain concerned about the limited scope of the theoretical results and the quality of the manuscript. I would like to maintain my original score.

Reviewer_1KRmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KW-DRO, a distributionally robust backup for cooperative value factorization under sparse actuator faults. Experiments validate the optimization components and investigate training stability and deployment performance in RWARE.

Critical strengths
  • The submodularity argument and its transfer to a robust-backup approximation bound appear correct under the stated assumptions.
Critical weaknesses
  • Figure 1 illustrates how agents’ actions can interfere with one another, but it does not clearly demonstrate uncertainty in the environment’s transition dynamics. Such action conflicts are a standard coordination issue in MARL; therefore, the example alone does not adequately motivate a formulation based on transition uncertainty.
  • There is no evidence why Wasserstein Ambiguity set is a proper form of the uncertainty set. Please add a discussion to compare with other forms of uncertainty set.
  • The benchmark and baselines are limited. Please add other robust MARL with transition uncertainty as baselines.
Questions for authors
  • In the problem formulation, is it equivalent to consider the fault vector as part of the new state, or append it to construct a new state.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 1KRm

Thank you for verifying our submodularity derivation and backup bound, and for four sharp questions on modeling and baselines. Your question on appending faults to the state motivated a new theoretical and empirical extension (Sec. 3.4, Sec. 5.3, Appendices H–I in revised PDF during rebuttal).

W1. Figure 1: action conflict vs. execution-induced transition uncertainty.

That reading of our original caption was fair, and we rewrote Fig. 1's caption and Sec. 1. Fig. 1 has no policy conflict: all four robots command FORWARD, which normally advances the convoy. The stall occurs in actuation when the lead drive fails and executes noop.

When ξ∈{0,1}N\xi\in\{0,1\}^N overwrites failed commands with noop to form executed action aξa^{\xi}, the state evolves under P0(⋅∣s,aξ)P_0(\cdot\mid s,a^{\xi}). Rather than an unstructured (s,a)(s,a)-rectangular kernel set, which can include physically unreachable transitions, our set restricts uncertainty to kernels reachable by at most KK simultaneous stalls — the multi-agent, KK-sparse counterpart of action-robust MDPs (Tessler et al., ICML 2019; Sec. 2, App. D). Because nominal rollouts never encounter stalls, nominal backups can overvalue tightly packed queues whose value drops sharply when one robot stalls.

W2. Why a Wasserstein set vs. alternative uncertainty sets.

Around the fault-free nominal P^=δ0\hat{P}=\delta_0, standard divergence sets break down:

  • KL / χ2\chi^2: Require P≪δ0P\ll\delta_0. Any mass on ξ≠0\xi\neq 0 gives D(P ∥ δ0)=∞D(P\,\|\,\delta_0)=\infty, collapsing the ball to {δ0}\{\delta_0\}.
  • TV / ρ\rho-contamination: Bound only P(ξ≠0)P(\xi\neq 0). Stalling KK actuators costs the same as stalling one, so the adversary puts all mass ρ\rho on the worst KK-set.
  • Support-constrained Hamming-W1W_1: Charges ∥ξ∥1\|\xi\|_1 per stalled robot, so W1(P,δ0)=EP[∥ξ∥1]W_1(P,\delta_0)=\mathbb{E}_P[\|\xi\|_1] (Lemma 5). Pairing this with supp(P)⊆ΞK\text{supp}(P)\subseteq\Xi_K separates concurrency (KK) from expected stalls/step (ρ\rho), while keeping allocation a 2D lower-hull LP (Theorem 7). Unequal per-actuator costs keep W1W_1 linear and are solved exactly by Alg. 2 (App. C.10); a non-Dirac nominal breaks linearity (App. A.1). Added after Definition 1.

W3. Baselines with transition uncertainty.

  1. DrIGM (Qu et al., ICLR 2026) in Table 1: Under DrIGM's vanishing-minimal-value condition, its ρ\rho-contamination target (Eq. 8 of arXiv:2602.11437) is r+γ(1−ρ)E[max⁡a′Qtot(s′,a′)]r+\gamma(1-\rho)\mathbb{E}[\max_{a'}Q_{\mathrm{tot}}(s',a')], identical for VDN to our scalar-shrinkage control r+(1−c)γVtot(s′)r+(1-c)\gamma V_{\mathrm{tot}}(s') at ρ=c\rho=c. Table 1 (10 seeds/layout, 10710^7 steps) compares KW-DRO with DrIGM-contamination (ρ∈{0.0036,0.0072}\rho\in\{0.0036,0.0072\}) and Double Q. At ρ=0.5\rho=0.5, KW-DRO keeps 20/20 runs alive vs. 14/20 nominal (p=0.020p=0.020) and 18/20 DrIGM-contamination (p=0.50p=0.50). (DrIGM's TV variant needs a learned dual network; Table 11 also contrasts action-robust RL and RADAR.)
  2. New 16-seed QMIX study vs. fault-randomized training (162 runs, 10710^7 steps; Sec. 5.3, App. I): We compared nominal QMIX, transient KW-DRO (ρ=0.5\rho=0.5), persistent KW-DRO (D=10D=10, dose-matched ρ=0.05\rho=0.05), and fault-randomized (FR) training (random D=10D=10 outages in rollouts) under targeted 20-step KK-robot outages (E1):
QMIX arm (16 seeds)tiny cleantiny E1small cleansmall E1
Nominal QMIX22.4 (3/16)19.726.7 (5/16)24.6
Fault-randomized (FR)24.0 (4/16)21.028.8 (6/16)28.0
Transient KW-DRO (ρ=.5\rho=.5)45.3 (11/16)35.727.0 (7/16)26.7
Persistent KW-DRO (D=10D=10)48.9 (11/16)38.036.0 (7/16)34.9

Deliveries/1k steps (competent seeds). Bold: paired p≤0.030p\le 0.030 vs. nominal. On tiny-4ag, KW-DRO roughly doubles throughput, and persistent KW-DRO beats FR on E1 (+17.0+17.0, p=0.005p=0.005). On small-8ag, persistent KW-DRO leads in mean throughput with wider seed variance (+10.3+10.3 vs. nominal on E1, CI [−1.5,21.9][-1.5,21.9]).

Q. Equivalence to appending the fault vector to the state.

For 1-step stalls (App. A.1), KW-DRO is the robust backup on (s′,ξ)(s',\xi) with adversarial ξ∼P∈Pρ,K\xi \sim \mathcal{P} \in P_{\rho, K}, except (i) a 1-step stall leaves no trace after noop, so V(s′,ξ)=max⁡a′∈A(ξ)Q(s′,a′)V(s',\xi)=\max_{a'\in\mathcal{A}(\xi)}Q(s',a') needs no ξ\xi input to QQ and healthy greedy actions stay independent of ξ\xi (Lemma 6); and (ii) fixed-law ξ\xi reduces to FR training. For persistent outages (D≥2D\ge 2) (Sec. 3.4, App. H), per-robot timers z∈{0,…,D}Nz\in\{0,\dots,D\}^N must join the state (s,z)(s,z) (each robot sees only its own ziz_i). This operator recovers D=1D=1, remains a γ\gamma-contraction (Prop. 13), preserves O(NK)O(NK) submodular greedy + hull allocation on healthy robots (Lemma 14), and activates KK even at ρ≤1\rho\le 1 as outages accumulate (Prop. 15, Example 16, Fig. 15d).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their time and constructive feedback. We are glad that the reviewers found concurrent actuator failures practically relevant and well motivated (9psG), confirmed that our submodularity derivation and its transfer to the robust backup hold under the stated assumptions (1KRm), and found the submodular formulation and its gains over nominal training promising (oMRc).

What is new. Prior robust (MA)RL robustifies the transition kernel (DrIGM) or trains against a learned adversary (action-robust RL, RADAR). KW-DRO shows that the worst-case backup over all ∑k≤K(Nk)\sum_{k\le K}\binom{N}{k} concurrent-fault patterns can be solved inside standard value factorization, with no learned adversary or transition model: under convex monotone QMIX mixing, fault damage is monotone submodular (O(NK)O(NK) greedy, (1−1/e)(1-1/e) guarantee, exact for VDN), and probability allocation is an exact lower-hull computation. The same structure carries over to persistent outages.


We have uploaded a revised PDF with all updates marked in blue (main text within the 10-page limit for rebuttal, plus expanded Appendices G–I). Key updates include:

  1. Execution-fault channel and Hamming-W1W_1 budget (Sec. 1, Sec. 3.1).

    A fault pattern ξ∈{0,1}N\xi\in\{0,1\}^N overwrites stalled robots' commands with noop, so the system steps under P0(⋅∣s,aξ)P_0(\cdot\mid s,a^{\xi}). Around the fault-free nominal P^=δ0\hat{P}=\delta_0, KL and χ2\chi^2 balls collapse to {δ0}\{\delta_0\} (infinite divergence outside nominal support), while TV and ρ\rho-contamination balls charge KK simultaneous stalls the same as one. Under the Hamming ground cost, W1(P,δ0)=EP[∥ξ∥1]W_1(P,\delta_0)=\mathbb{E}_P[\|\xi\|_1] (Lemma 5) charges per failed actuator: ρ\rho budgets expected stalls per step, while supp(P)⊆ΞK\text{supp}(P)\subseteq\Xi_K caps simultaneous stalls per step at KK. This budget view now appears directly after Definition 1.

  2. Comparison and exact baseline equivalence with DrIGM (Sec. 2, Sec. 5.2, Table 1, App. D).

    To clarify, we left short sentences and a separate App. D in our initial manuscript describing the difference between our work and DrIGM. Now, we expanded Sec. 2 and App. D (Table 11) and added an algebraic link: under DrIGM's vanishing-minimal-value condition, its ρ\rho-contamination target (Eq. 8 of DrIGM) is r+γ(1−ρ)E[max⁡a′Qtot(s′,a′)]r+\gamma(1-\rho)\mathbb{E}[\max_{a'}Q_{\mathrm{tot}}(s',a')], which for VDN coincides with our scalar-shrinkage backup r+(1−c)γVtot(s′)r+(1-c)\gamma V_{\mathrm{tot}}(s') at ρ=c\rho=c. Table 1 thus directly evaluates DrIGM-contamination (ρ∈{0.0036,0.0072}\rho\in\{0.0036,0.0072\}, 10 seeds/layout) at matched mean pessimism.

  3. Multi-step persistent outages via state timers, activating KK at ρ≤1\rho\le 1 (Sec. 3.4, App. H).

    Prompted by Reviewer 1KRm's state-augmentation question, we extended KW-DRO to outages lasting D≥1D\ge 1 steps by adding per-robot timers z∈{0,…,D}Nz\in\{0,\dots,D\}^N to the state (App. H, Props. 13–15, Lemma 14). The operator recovers our transient backup at D=1D=1, remains a γ\gamma-contraction on (s,z)(s,z), and preserves O(NK)O(NK) submodular greedy selection plus exact lower-hull allocation on healthy robots. Because outages started on different steps accumulate against KK, KK shapes the fixed point even when ρ≤1\rho\le 1 (raising KK from 1 to 4 at ρ=0.5,D=10\rho=0.5, D=10 lowers the exact value by 61%61\%, Example 16; on trained networks at ρ≤1\rho\le 1, moving KK from 1 to 2 changes the target on 87%87\%/74%74\% of states with an outage in progress, Fig. 15d).

  4. New 16-seed QMIX study (162 runs at 10710^7 steps) vs. nominal QMIX and fault-randomized training (Sec. 5.3, App. I).

    On tiny-4ag (K=2K=2, 16 seeds/arm), both transient KW-DRO (ρ=0.5\rho=0.5) and persistent KW-DRO (D=10D=10, dose-matched ρ=0.05\rho=0.05) more than double fault-free throughput over nominal QMIX (45.3 and 48.9 vs. 22.4 deliveries/1k steps; p=0.019p=0.019 and p=0.006p=0.006; 11/16 vs. 3/16 competent seeds), and maintain large gains under targeted 20-step KK-robot outages (35.7 and 38.0 vs. 19.7; p≤0.030p\le 0.030); persistent KW-DRO also beats fault-randomized training under these outages (38.0 vs. 21.0; p=0.005p=0.005). On small-8ag (K=3K=3), persistent KW-DRO leads in mean throughput under outages (34.9 vs. 24.6 nominal and 28.0 fault-randomized), though the paired difference is not resolved at 16 seeds (+10.3+10.3, CI [−1.5,21.9][-1.5, 21.9]). On trained networks, greedy matches exhaustive search within 1.4×10−51.4\times 10^{-5} on 2,904 DD-step cases (Fig. 15b), with worst ratio 0.9998 over both stall types (Table 16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.