acceptodds
Under review as a conference paper at ICLR 2027

SIR-MARL: Post-Training Recovery from Group-Sparse Observation Faults in Multi-Agent Reinforcement Learning

Abstract

Cooperative multi-agent policies trained with nominal observations can encounter sensor failures and perception errors after deployment. We study post-training observation recovery for frozen recurrent policies using clean trajectories, with unknown fault support and cardinality. Shared task dynamics produce related observations across agents, and recent history constrains their admissible clean values. Through a synchronized team-level recovery layer, SIR-MARL exploits this history-conditioned low-dimensional structure together with corruption that is sparse across agent groups. It predicts clean observations from temporal and cross-agent evidence, uses group residuals to guide correction, and refines recovery through leave-one-out contexts. Generic group masking and response-based supervision train the two-stage recovery process on nominal trajectories without matched deployment-fault data. Under structural, observability, and residual-separation conditions, our analysis bounds recovery error and its effect on the frozen actor's responses. Experiments on three StarCraft Multi-Agent Challenge (SMAC) tasks show improved faulted performance while largely preserving nominal performance. Structural and mechanistic diagnostics relate these gains to history-conditioned observation structure, sparse-error correction, and the quality of refined context.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, robust MARL, observation recovery, sparse recovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iBTzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses post-training robustness in multi-agent reinforcement learning (MARL) policies, where observations may be noisy due to faulty sensors. It proposes a recovery mechanism, called SIR-MARL, which uses the joint observation of all agents to infer group perceptions errors in order to reconstruct valid observations. SIR-MARL is evaluated in three SMAC tasks and compared with some baselines.

Critical strengths

The paper addresses a practically relevant problem in cooperative MARL.

Critical weaknesses

Problem formulation

While the problem is important, it is not new and already known from distributed systems [1,2,3]. However, the introduction does not contain many references (the first paragraph does not even have any), which is important for scientific integrity.

The paper focuses on Dec-POMDP, which already features noisy observations [2,4,5]. SMAC, on the other hand, represents a special case of Dec-POMDPs with deterministic observations and initial states [5,6]. Thus, the paper seems to specialize more on the SMAC setting rather than the general Dec-POMDP.

Limited significance

On the one hand, the paper assumes noisy observations at inference time, which is a realistic assumption. However, the recovery mechanism relies on the joint observation, which requires perfectly reliable global communication, which is rather unrealistic [1,2].

I also have concerns regarding the evaluation:

  • SMAC is known to be outdated (due to its special case traits [5,6]).
  • Only three tasks have been presented, despite SMAC having more than 10 different maps.

For a better evaluation, it might make sense to use SMAC as an idealistic training setup, while using SMACv2 [6] as the test setup (and demonstration of the recovery method).

Literature

[1] Tanenbaum et al., "Distributed Systems: Principles and Paradigms"

[2] Oliehoek et al., "A Concise Introduction to Decentralized POMDPs", 2016

[3] Stone et al., "Multiagent Systems: A Survey from a Machine Learning Perspective", Autonomous Robots 2000

[4] Lyu et al., "A Deeper Understanding of State-Based Critics in Multi-Agent Reinforcement Learning", AAAI 2022

[5] Phan et al., "Attention-Based Recurrence for Multi-Agent Reinforcement Learning under Stochastic Partial Observability", ICML 2023

[6] Ellis et al., "SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning", NeurIPS Benchmarks 2023

Questions for authors

None

Reviewer_ZiLdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies deployment-time observation faults in cooperative MARL. A frozen recurrent policy, trained on clean observations, receives observations in which an unknown subset of fewer than half the agents is densely corrupted. SIR-MARL adds a trusted, centralized recovery layer in front of the frozen actors, working in two stages:

  • Stage 1 predicts clean observations from recovered history and cross-agent attention. It scores each agent's residual against a median/MAD (median absolute deviation) team threshold, and softly replaces suspicious agents' observations.
  • Stage 2 refines each target from two leave-one-out contexts, one built from the raw input and one from the Stage-1-corrected input, then repeats the residual-gated correction.

Training uses only clean trajectories. Synthetic corruption comes from random group masking (replacing agents' observations with feature means), plus "decision-aware" losses that pass gradients through the frozen actor to keep its action distribution close to the clean one. The theory gives a bound on the dimension of history-conditioned observation sets (Prop. 5.1), a cancellation error bound (Thm. 5.2), a Stage-2 sensitivity bound (Thm. 5.3), and a finite-horizon bound on how recovery error affects the actor's actions (Cor. 5.5). On SMAC 8m, 3s5z and 2s3z under per-step Gaussian faults on s = 1–4 agents, SIR-MARL keeps near-source clean performance and substantially improves faulted win rates over corrupted MAPPO and five robust-MARL baselines (Table 1). It also helps on GRF and MPE (App. D). Appendices add fault-family transfer tests, an augmentation baseline, and paired source-level intervals.

Critical strengths

1- A practical setting, plus clear evidence that decision-aware recovery matters: Robustifying a frozen policy after training, using only clean data, with unknown fault support and cardinality, is a realistic deployment need. The decision-aware objective is well supported:

  • Removing the decision losses drops 8m s4 from 35.4% to 16.5% (Table 2).
  • Direct-actor supervision beats a fitted surrogate, which in turn beats no decision loss, on all three maps (Table 10).
  • Table 11 shows that lower reconstruction MSE does not imply better action preservation. On 8m, the no-decision variant has lower MSE but higher actor KL and worse win rate. This is a useful, transferable finding for anyone building observation filters in front of a fixed policy.

2- Thorough and honest experimental reporting: The appendices report failure cases openly: 2s3z beyond a clean majority, fault families where gains vanish (Table 28), and the augmentation baseline outperforming the method (Table 31). They also include oracle-support and known-cardinality controls (Table 27), inference and training sensitivity sweeps, compute profiling, cross-domain GRF/MPE results, and source-paired intervals (Table 32). This makes the paper's claims checkable, which is more than many robust-MARL papers offer.

Critical weaknesses

1- The robustness evidence comes from one easy-to-detect fault model, and the paper's own transfer tests contradict the generality claim.

  • The main benchmark is favorable: All main results use isotropic Gaussian noise at three times the team observation's standard deviation, with the corrupted agents re-drawn every step (App. C.1.2). Such faults are gross outliers, and earlier clean frames of a faulted agent are usually available. Detection is essentially solved: support F1 is 99.9–100% on 8m, and estimated support gives the same win rates as oracle support (Table 27).
  • Other fault types are not recovered: With persistent supports (App. F, Fig. 3b, Table 28), the method recovers meaningfully only for F0 (the same 3× Gaussian) and F1 (feature-mean replacement, which is exactly the training corruption). For realistic-scale noise (F2), fixed bias (F3), stale observations (F4) and nominal-pool replacement (F5), support F1 is 0.00–0.13 and gains are between −4.9 and +2.6 points across all three maps.
  • The paper motivates the work with "sensor failures and perception errors" whose characteristics are unknown in advance, and claims recovery "without matched deployment-fault data". The evidence supports something narrower: recovery from large-magnitude outliers or faults resembling the training mask.

2- The baselines don't isolate the contribution, and the most relevant baseline beats the method.

  • The main baselines target a different problem: The baselines in Table 1 were designed for adversarial observation or action attacks (ROMANCE and WALL train against action attacks). They are evaluated off-target on Gaussian observation noise, and all are decentralized. SIR-MARL instead uses a centralized layer that receives all nD observation scalars every step (App. C.1.1), which the main text mentions only briefly.
  • The augmentation baseline wins: The most relevant control is SparseAug-Mix, which trains MAPPO with Gaussian observation augmentation. It appears only in App. G.3 (Table 31) and outperforms SIR-MARL averaged over s1–s4 by 14.4 points on 8m, 7.2 on 3s5z and 30.5 on 2s3z (e.g. 2s3z s3: 72.5% vs 13.2%). The text describes this only as "higher at s3 and s4". SIR-MARL does preserve clean performance better (8m: 83.3% vs 77.2%), which is a real advantage, but the trade-off should be stated in the main text.
  • The same-interface baselines are weak: PCA-SIR performs worse than doing nothing (normalized gap recovery of −11.7%, −6.0% and −23.5%, Table 33; GRF s1: 1.0% vs 36.0%, Table 16), which suggests a misconfigured baseline. -The two-stage design isn't clearly supported. The one-stage SIR-Direct model is within noise of SIR-MARL on 3s5z (+1.94, CI [−1.28, +5.84]) and significantly better on 2s3z (−1.22, CI [−2.11, −0.17]) (Table 32). So the main architectural contribution is supported only on 8m.
  • A simple imputation baseline is missing: Because detection matches the oracle, a simple imputation baseline for detected agents is needed to show what the architecture adds. Examples: holding the last recovered observation, a linear or Kalman temporal predictor, or a masked-autoencoder imputer with the same median/MAD gate.
Questions for authors

1- Can you add a simple imputation baseline using the same median/MAD gate, i.e. hold-last-recovered-value or a linear/Kalman temporal predictor for detected agents? Given that estimated support equals oracle support on 8m (Table 27), this would show how much the two-stage architecture adds beyond detection. A clear margin would address part of Weakness 2.

2- How do performance and support F1 change as the Gaussian scale drops from 3× toward 0.5× the team standard deviation, with persistent supports? Where does residual detection break down, and how does the actor's sensitivity at that scale compare?

3- Why does PCA-SIR fall below corrupted MAPPO, and on GRF to about 1% win rate at s1, when it "preserves observations outside the estimated support"? Is its threshold or rank misconfigured for heterogeneous, non-Gaussian observation features?

4- How should the main comparison be read given that SparseAug-Mix outperforms SIR-MARL in most faulted cells (Table 31)? Would SIR-MARL on top of a SparseAug-Mix policy combine the advantages?

5- The diagnostics show the theory's sufficient conditions don't hold on the evaluated inputs. The recurrent Jacobian is non-contractive (gain below 1 in only 0–25% of samples), and the clean-side interval margin is nonpositive (App. C.5). Which theoretical statements actually apply to the reported experiments?

1 comment
Reviewer_ZiLdOpenReview ↗

Response to rebuttal

Thank you for the new experiment. I have updated my assessment based on the most recent provided information.

Reviewer_fJAXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposed SIR-MARL, a two-stage module that repairs corrupted observations before passing them to a frozen multi-agent policy. It uses recent history and teammates observation, training on clean trajectories with marking and supervision that preserves the policy's decisions. The authors provide conditional recovery bounds and experiments on SMAC, Google Research Football, and MPE, with strong gains under the main gaussian fault settings.

Critical strengths

The frozen-policy concept in practically relevant, and the improvements over the unprotected policy are substantial. On SMAC, the 8m environment with two corrupted agents recovered from 12% to 60% with a small change in the clean performance. For a MARL task, achieving this by using clean trajectories and generic masking is very useful.

The decision-supervision ablations are informative. Table 2 states that removing some these losses substantially reduces performance under increasing corruption, which supports the choice to train recovery around the policy's decisions.

The additional tests shown on figure 3 makes the methods limitations more explicit.

Critical weaknesses

For me looks like the strongest robustness results comes from easily detectable observation errors. The main Gaussian protocol may produces almost perfect fault localization in several settings, whereas stale observations and nominal replacement receive little benefit and sometimes become worse after recovery (as shown in figure 2b). From my point of view this limits drastically the scope of the findings, since the evidence is stronger for correcting outliers rather than handling unknown observation faults more generally.

The added value of the two-stage architecture is not consistently established against the closest matched baseline. Compared with SIR-Direct, SIR-MARL improves average faulted win rate by 7.5% on 8m env, but only 1.9% on 3s5z and performs worse on 2s3z. Since the baseline shares the frozen-policy and clean-only training setup, it raises questions about how much the proposed refinement contributes.

Questions for authors

How do SIR-Direct and SIR-MARL compare in parameter count and inference cost? is there evidence that the improvement on 8m comes from the refinement mechanism rather than an additional model capacity?

For the stale observations and nominal replacement, is the main problem missed fault detection or inaccurate reconstruction? separating these two things would help explain faults the method can address.

Comments on the paper

2 comments

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

AuthorsOpenReview ↗

Summary of Manuscript Revisions

We summarize the additional experiments and clarifications in the revised manuscript below.

1. Baseline roles and architecture comparisons (fJAX and ZiLd).

Section 6 introduces SIR-Direct and SparseAug-Mix as additional comparisons, with configurations and results in Appendices G.2 and G.3. SIR-Direct is an alternative architecture within our proposed recovery framework, retaining its core mechanisms with a comparable parameter and FLOP budget. SparseAug-Mix relaxes the frozen-policy and unknown-fault constraints by updating the policy using synthetic faults from a Gaussian family specified before training. Section 6 also explains the inclusion of ROMANCE and WALL as tests of transfer from their training mechanisms to observation faults, with comparison protocols in Appendix C.2.

2. Corrected PCA-SIR results (ZiLd).

We have replaced the PCA-SIR results in Table 1 and the GRF comparison in Table 17 using the centered configuration, and updated the source-normalized statistics in Appendix G.4. The full evaluations use five source policies with 64 episodes per source and condition on SMAC, and three source policies with 500 episodes per source and condition on GRF. Appendices C.2 and D.2 specify the configuration. Appendix G.7 retains the scaling, rank, and threshold comparisons using three source policies and 64 episodes per source and condition. Section 6.3 explicitly locates the observation that centered PCA approaches SIR-MARL on GRF in the matched s1 configuration comparison.

3. Fault scope and refinement gains (fJAX and ZiLd).

Section 6.1 and Appendix F describe the limited recovery under stale observations and nominal-pool replacements together with their low support recall. Appendix F explains how distinguishing these replacements depends on the conditional information supplied by history and teammates, and relates the measurements to Appendix A.6's identifiability analysis. Appendix G.2 reports the cross-task architecture comparison: SIR-MARL has a clear advantage on 8m, while SIR-Direct performs similarly on most evaluated tasks and is better in some conditions.

4. Additional experiments, statistical reporting, and theoretical interpretation (fJAX and ZiLd).

Appendix F.1 reports persistent two-agent Gaussian scale sweeps and variance-matched uniform faults on 8m, including source-paired confidence intervals and actor-response diagnostics (Tables 30-31). The sweep scales noise by the clean team's standard deviation; F2 uses feature-specific scales. All Gaussian scales have positive mean win-rate differences, with the 95% interval above zero at scale 1. The actor probes also report increased KL after recovery at scales 0.5 and 1. Uniform noise at scale 1 yields a gain of 15.63 percentage points, with a 95% interval of [11.74, 19.51].

Appendix G.6 reports closed-loop Hold-last and Linear comparisons, including clean-input false positives, alongside offline comparisons holding the SIR gates, history, and actor context fixed (Tables 39-45). The fixed-gate comparisons provide direct evidence of candidate quality; the closed-loop results assess each method's combined prediction, gating, and history updates. Appendix C.2 specifies the source-level paired statistics. For these new closed-loop comparisons, each condition uses three source policies and 64 episodes per source and method, with Student-t intervals computed from the three paired source-level differences.

For ZiLd's theoretical question, Appendix C.5 explains how realized residual margins enter the recovery bounds and how the interval condition provides a sufficient certificate. It also relates the noncontractive Jacobian measurements to the finite-horizon actor-response bound in Appendix A.14, which allows gains above one.

5. Background and supporting references (iBTz).

Section 1 now includes references supporting the opening discussion of nominal policy training, deployment-time observation failures, and structured recovery. We have also supplemented references for related background statements elsewhere in the manuscript.

We thank reviewers fJAX and ZiLd for their follow-up feedback on our response and have incorporated the resulting clarifications on baseline comparisons, fault scope, and theoretical conditions into the revised manuscript. We would also welcome reviewer iBTz’s feedback on our response and the revised manuscript, particularly on which aspects of the problem setting, communication assumptions, or evaluation would benefit from further clarification or a specific comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.