acceptodds
Under review as a conference paper at ICLR 2027

When the Safety Signal Lies: Corrupted Cost Feedback in Constrained Multi-Agent RL

Abstract

Safe multi-agent reinforcement learning assumes that the cost signal driving constraint enforcement is reported faithfully, yet decentralized deployments receive it over a network as cost reports, an attack surface. We first show that for any connected doubly stochastic mixing matrix, an unclamped fixed point of consensus dual ascent constrains only the network average of the agents' cost estimates, not each agent's. We then establish corruption mass conservation: along a fixed primal sequence, the unprojected consensus recursion preserves the aggregate dual bias that corrupted reports inject, so topology can only redistribute it. A biased multiplier scales the constraint gradient, whereas a reward perturbation that shifts every return equally leaves the policy gradient unchanged. We propose Robust Constraint Estimation (RCE), which trims each agent's cost reports and adds a dispersion margin; under an honest-source majority it bounds the estimation error and, unlike gating updates on the untrimmed mean, cannot be stalled by over-reporting. In one instrumented testbed, persistent under-reporting from one of three sources raises true network-average cost by against a budget of (paired CI ) while the reported cost falls; a matched zero-mean perturbation does not reproduce the effect, an unconstrained control shows that the constraint holds cost near budget, and the effect increases across the tested doses . Within each block RCE removes of the effect at , where it reduces to the median, and at , where its margin is live; the margin adds clean-run conservatism but no measurable suppression, and residual harm remains. Finally, corrupting a single agent's reports, the case the spreading result addresses, yields the intended concentrated perturbation, but the realized multipliers do not follow the fixed-primal prediction: displacement is not localized on the attacked agent, and a median-deviation monitor pre-declared to catch the attack without communication detects nothing under either topology. Empirical claims are descriptive: one testbed, one operating point, three seeds.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Multi-Agent Reinforcement Learning, Safe Reinforcement Learning, Adversarial Robustness, Reward and Cost Poisoning, Decentralized Reinforcement Learning, Primal-Dual Optimization, Robust Aggregation, Byzantine-Robust Learning, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RioGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies corrupted cost feedback in constrained multi-agent reinforcement learning. It analyzes how biased reports affect dual updates and proposes RCE, which combines trimmed aggregation with a conservative margin based on median absolute deviation. Experiments on six-agent ManySegmentAnt, using three seeds and several attack and control settings, show that persistent underreporting increases true cost. RCE reportedly reduces the attack-induced cost increase by approximately 80%. Additional experiments examine how corruption propagates through dual mixing and whether it escapes detection.

Critical strengths

The paper addresses a meaningful failure mode: corrupted constraint feedback can weaken safety enforcement without directly changing rewards or policy observations. This is a useful perspective on the robustness of constrained multi-agent learning.

The experiments provide an initial demonstration of the vulnerability and a substantial reduction in attack-induced cost under robust aggregation. The released code and detailed logs also make the reported behavior inspectable, providing a useful basis for further investigation.

Critical weaknesses
  1. Theorem 5 overlooks corrupted reports surviving trimming and the selection bias introduced among retained honest reports, leaving its error bound unsupported. Proposition 7 and the empirical MAD calibration do not resolve this gap.

  2. Eq. (2) omits each policy’s effects on other agents’ costs, while training and cost estimation handle termination differently. The mechanism gap also compares reports and evaluations from different policies, conflating reporting error with policy changes.

Questions for authors
  1. Provide a corrected proof of Theorem 5 that accounts for corrupted reports surviving trimming and the selection bias among retained honest reports? Which guarantees for RCE and Proposition 7 remain valid under the corrected bound?

  2. What precise constrained objective does the implementation optimize? How is Eq. (2) justified when an agent’s policy affects other agents’ costs, and how are the different termination treatments in GAE and cost estimation reconciled?

  3. Do the mechanism-gap conclusions hold when reports and oracle evaluations use the same frozen policy?

Reviewer_fUKHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates corrupted cost feedback in decentralized constrained MARL and proposes REC using trimmed aggregation and an MAD-based safety margin. The problem is relevant, and the paper provides useful empirical observations. However, I find the theoretical contribution limited, the methodological novelty insufficiently established, and the experimental evidence too narrow to support the broader motivation.

Critical strengths

The paper identifies an interesting attack surface in safe MARL. Its experiments demonstrate a meaningful discrepancy between reported and true safety costs, and the authors are transparent about the limitations of their findings.

Critical weaknesses
  1. Limited theoretical contribution and practical relevance. Theorem 2 largely follows from the doubly stochastic property of the mixing matrix under fixed-primal and unprojected assumptions. Its relevance to the actual closed-loop MARL system remains unclear, especially since Section 7.5 does not confirm the predicted multiplier behavior. Furthermore, Proposition 1 identifies a per-agent versus network-average constraint mismatch that RCE does not address.

  2. Limited methodological novelty and experimental validation. RCE mainly combines standard trimmed-mean aggregation with an MAD-based margin. At (M=3), it reduces to median aggregation, while at (M=5), the margin provides no measurable additional attack suppression. The experiments primarily consider uniform, persistent under-reporting, which resembles a simple budget shift, and are limited to one environment and three seeds. Comparisons with standard robust aggregation baselines are missing.

Questions for authors

Q1. Given the restrictive assumptions of Theorem 2 and the discrepancies observed in Section 7.5, what additional insights do the theoretical results provide for the actual closed-loop MARL system? How does Proposition 1 motivate RCE when the proposed method does not address per-agent constraint satisfaction?

Q2. What is the methodological novelty of RCE beyond existing Byzantine-robust aggregation techniques? Can the authors provide direct comparisons with standard trimmed-mean aggregation and ablations of the MAD-based margin?

Q3. Can the authors demonstrate that the observed attack and defense effects extend beyond an effective budget shift under uniform under-reporting, particularly for localized or less conspicuous attacks and across additional environments?

Reviewer_U6GUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether safety constraints in decentralized multi-agent reinforcement learning can be undermined by corrupted cost reports. In this setting, each agent updates a penalty weight (multiplier) for its own cost constraint using cost estimates aggregated from multiple reporting sources. The authors formalize a constrained Markov game with corrupted cost feedback and establish several structural properties: at a fixed point of the specified consensus dual update with no multiplier at its upper cap, the update constrains the network average of the reported cost estimates rather than each agent's estimate separately; along a fixed primal sequence, while projection is inactive, the aggregate corruption-induced dual bias is conserved, with topology only redistributing it; and a policy-independent offset to the reward return leaves the exact policy gradient unchanged, whereas a cost-report offset scales the constraint gradient. They propose Robust Constraint Estimation (RCE), which trims extreme reports and adds a dispersion margin, and present a bounded-error analysis under an honest-source majority and a liveness result for over-reporting. The empirical study uses one simulated multi-agent robot testbed and reports that sustained under-reporting can increase the true network-average cost while reducing the reported cost, that RCE reduces but does not eliminate this effect, and that a pre-declared detection rule fails to identify a concentrated attack.

Critical strengths
  1. The paper identifies a concrete and well-motivated integrity problem: decentralized constrained learners act on reported cost estimates rather than directly measured costs, meaning that the corruption of these reports can directly affect constraint enforcement. It further clarifies why this is not merely reward poisoning in the analyzed dual-mediated channel, where the cost report drives a dual variable that scales the constraint gradient, whereas a policy-independent offset to the reward return does not affect the exact policy gradient. This is consequential because it makes an implicit assumption of safe-MARL evaluation explicit and testable: if cost reports can be corrupted, reported safety metrics may not reflect actual constraint violations.
  2. The paper is transparent regarding its evidence. The main experimental campaigns are pre-declared, several controls are included (a zero-mean falsification control, an unconstrained control, and a dose-response study), negative results are reported, and artifacts are described in detail. This enhances the credibility of the findings and provides a reasonable basis for follow-up work.
Critical weaknesses
  1. The contribution of the dispersion margin could be further clarified. At M=3M=3 and f=1f=1, RCE reduces to median aggregation with a negligible margin. Although the margin becomes active at M=5M=5, the experiments lack a matched β=0\beta=0 ablation, making it difficult to distinguish its contribution from that of trimming. A controlled comparison under identical settings and seeds would help establish whether the margin provides additional attack suppression or clean-run conservatism beyond standard trimmed-mean aggregation [1].

  2. The empirical evaluation could be further extended. The experiments focus on a single ManySegmentAnt environment, three seeds, and persistent under-reporting attacks with f=1f=1. Moreover, the reporting sources are simulated through parallel rollout batches, leaving their correspondence to real-world reporting components unclear. Since only the controller in Eq. (2) is evaluated, testing additional attack regimes and another safe MARL method, such as MACPO [2], would help demonstrate the broader applicability of the findings.

[1] Yin, D., Chen, Y., Ramchandran, K., and Bartlett, P. "Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates." International Conference on Machine Learning (ICML), 2018.

[2] Gu, S., Grudzien Kuba, J., Chen, Y., Du, Y., Yang, L., Knoll, A., and Yang, Y. "Safe Multi-Agent Reinforcement Learning for Multi-Robot Control." Artificial Intelligence, 319:103905, 2023.

Questions for authors
  1. Can the authors provide a matched M=5 comparison between the full RCE method and its β=0 counterpart under identical settings? This would isolate the margin's specific contribution to attack suppression and clean-run conservatism from that of trimming.
  2. What do the reporting "sources" correspond to in a practical deployment, and under what access constraints would an adversary control one source while leaving others uncompromised?
  3. The theoretical analysis is specific to the controller in Eq. (2). Which analytical assumptions are essential for generalization to other decentralized constrained learners (e.g., doubly stochastic consensus, an unclamped fixed point, or inactive projection), and which are specific to the evaluated implementation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.