acceptodds
Under review as a conference paper at ICLR 2027

Adversarially Robust Multi-Agent RL via Test-Time Imagination

Abstract

Recent advances in model-free multi-agent reinforcement learning (MARL) have improved policy robustness against adversarial observation perturbations at test time. However, training such robust policies requires extensive interaction with the environment. Multi-agent world models could reduce the interaction cost by enabling robust policy learning from imagined experience. Multi-agent world models, however, typically condition their predictions on global information unavailable to individual agents at execution time, making them unsuitable for decentralized defense. Moreover, policies trained from imagined experience are not inherently robust to adversarial observation perturbations. We propose MATRIX, a world-model-based MARL framework that unifies sample-efficient policy learning with training- and test-time mechanisms for robustness to test-time adversarial observations. MATRIX applies classifier-free guidance to a diffusion-based world model that predicts each agent’s next observation using either local information alone or using local and global information jointly. This design enables MATRIX to use a single world model for three complementary functions: training policies on model-generated trajectories, robustifying policies using model-generated trajectories, and predicting observations from local information for decentralized test-time defense. Building on locally conditioned observation prediction, we propose a low-overhead execution protocol that uses the trained world model to mitigate the effects of adversarial observation perturbations at test time, without requiring a separate defense model. Evaluations in environments with discrete and continuous action spaces demonstrate that MATRIX outperforms state-of-the-art baselines in both sample efficiency and robustness to test-time observation attacks.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, world models, adversarial attacks, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8Ky6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents MATRIX, a framework built on a diffusion world model, aimed at sample-efficient multi-agent RL and at decentralized defense against observation attacks at test time. The key idea is that global conditioning is dropped selectively, thus a single model can be used three different ways: for centralized policy learning, for robustifying the policy against locally predicted observations, and for replacing observations at execution in a decentralized manner. Whenever the detector thinks an observation looks suspicious, replacement is triggered. The detector is a CUSUM test on the denoising loss. Experiments on SMAC and MaMuJoCo cover sample efficiency, adversarial robustness, and execution overhead.

Critical strengths

S1. Reusing one world model for both policy learning and decentralized observation defense is a sensible design, and the paper follows it through consistently. S2. The experiments cover both discrete and continuous action spaces, and the ablations give at least some support for the roles of policy robustification and test-time defense.

Critical weaknesses

W1. The proposed framework lacks novelty. Every component already exists, and the contribution is mainly in how they are wired together. And I could not tell what the robustification step buys you over simply training longer or training better. The paper never separates the two. W2. The experiments do not support the robustness claims as strongly as the paper suggests. The baselines are not matched in clean performance, the detector is not evaluated on its own terms, and the gap between what the policy sees during training and what it sees at execution is not tested.

Questions for authors
  1. I do not see much novelty beyond integration. The individual pieces are established techniques, and what is being claimed is essentially the architecture plus the training procedure.
  2. The world-model baselines fail, and the paper offers model distribution shift as the explanation. That may be right, but the experiments shown do not really establish it in every setting. It is asserted more than demonstrated.
  3. The detector assumes a harmful perturbation raises the locally conditioned denoising loss, yet no detection rate, no false-alarm rate, and no detection delay are reported. Appendix D.2 also chooses thresholds from FGSM and ATSA episodes, but never says what the numerical constraints on that choice were.
  4. Table 1 compares methods whose clean performance differs enormously. MATRIX reaches 5267 on Ant while the model-free robust baselines sit around 420–460. With starting points that far apart, a higher attacked return mainly shows better performance within the sample budget. It does not isolate greater resistance to perturbations.
  5. WM-ATSA and WM-PAAD are trained against frozen models that were learned under different policies, whereas MATRIX gets to alternate between policy learning and model updates. Their clean win rates are near zero, so the policies had not learned the task before robustness was even measured. These results do not isolate the advantage of MATRIX's robustification.
  6. During robustification the selected agents use their own predictions for the whole imagined rollout. At execution, however, corrupted observations the detector misses still enter the conditioning history, and agents flip between received and predicted observations according to CUSUM. This training–execution mismatch is never evaluated, even though the accuracy of later detection and replacement depends on it directly.
3 comments

We thank the reviewer for their comments. Our responses are provided below. We have revised the manuscript accordingly, with the main changes in the text highlighted in blue.

Weakness1, Question 1 (on novelty):

We agree that several building blocks, such as diffusion models and classifier-free guidance, are established. However, MATRIX contributes their adaptation and coordinated use for sample-efficient policy learning and decentralized robustness in MARL. Specifically, (i) we adapt classifier-free guidance to the CTDE information structure by dropping global conditioning while preserving each agent’s local context (local conditioning), allowing one shared world model to make both globally and locally conditioned predictions. This allows agents to perform imagination only based on their local information, a capability absent from existing world models; (ii) we introduce a mixed-mode imagination procedure that uses these local predictions to expose policies to prediction errors and their effects on coordination; and (iii) we propose a decentralized test-time defense that uses the model’s local denoising loss for anomaly detection and performs full diffusion sampling only when triggered. These methodological contributions together allow us to address an underexplored problem: sample-efficient learning of adversarial robustness in MARL.

Weakness 1, Question 4, Question 5 (on the gains of robustification & baselines with matched clean performance):

Regarding the benefits of MATRIX's robustification procedure compared to simply training for longer, we would first like to emphasize that the primary motivation of our work is sample efficiency. Accordingly, the most relevant comparisons are those conducted under a fixed, limited budget of interactions with the real environment. Under this constraint, existing model-free approaches struggle to learn effective policies, as demonstrated in Table 1.

This leaves two main alternatives for existing approaches. The first is to use a general-purpose world model, such as DIMA or MARIE, to learn a policy. These approaches can achieve competitive clean performance but lack robustness against test-time attacks. We had already compared the robustness of MATRIX against these methods, selecting the best-performing run of DIMA and MARIE, in Appendix E.1 of the original manuscript. However, we acknowledge that this comparison was not sufficiently detailed. To clarify the results, we have updated Table 8 in Appendix E.1 to explicitly report the percentage performance degradation under attacks relative to clean performance and added a clearer reference to this comparison in the main text. The results demonstrate that even when MATRIX and the baselines achieve comparable clean performance, MATRIX exhibits substantially greater robustness against adversarial attacks (e.g., a 4.2% performance drop for MATRIX versus 50% for MARIE in MMM against ATSA attack). Importantly, these approaches follow the same alternating model and policy update procedure and are trained under the same sample budget as MATRIX. Thus, the improved robustness can be attributed to the robustification-related mechanisms introduced by MATRIX during training and test-time.

The other possibility is to use the generative capability of a trained world-model and apply one of the model-free robustification approaches. Since the real sample budget has already been consumed to train an accurate world model, the training of the model-free approaches must rely exclusively on imagined trajectories generated by the frozen world model. Although these methods can use an unlimited number of imagined samples, the results for WM-ATSA and WM-PAAD in Table 1 show that this approach also falls short in terms of robustness.

Finally, to provide a more comprehensive evaluation, we have added comparisons between MATRIX and the model-free baselines trained with substantially larger sample budgets. Figure 8 in Appendix E.2 of the revised manuscript presents these results, including settings where the methods achieve comparable clean performance. The results show that, Compared to robust methods (ATSA, PPAD), MATRIX remains competitive in many cases, even when these model-free baselines are given significantly more training samples (e.g., see 2s3z in Figure 8). The figure also includes MAPPO, a non-robust baseline, which exhibits substantially greater performance degradation under adversarial attacks compared to robust methods (e.g., approximately 62.5% versus 9.8% for MATRIX in 2s3z) . This further confirms that longer training alone does not necessarily lead to improved robustness.

Thus, in summary, (1) Longer training alone does not guarantee robustness for algorithms without explicit robustification mechanisms, and (2) MATRIX achieves robustness comparable to that of existing robustification methods while requiring substantially fewer real-environment training samples.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Question 2:

We have extended the analysis in Appendix B of the revised manuscript. Table 4 reports, for all six scenarios, the loss ratio between ATSA and the co-trained policy and the real-versus-simulated reward gap, with the metrics defined in the experimental setup. Figure 5 shows the MaMuJoCo reward comparison and complements the existing SMAC results (Figure 4). These results show that in each scenario, at least one measure is considerably different for ATSA compared to the co-trained policy. It is not straightforward to isolate the effect of the world model-policy mismatch, since the policies visit different trajectories. However, these results indicate that the world model generalizes poorly to unseen policies in all considered scenarios.

Question 3:

We have added the evaluation of MATRIX's detector performance in Table 3 (for newly-added adaptive attacks), and in more detail in Table 9 and Figure 9 in Appendix E.3 of the revised manuscript for other attacks. The results show that, for example, in 2s3z, MATRIX's detector can identify even small perturbations with a budget of 0.05 with a detection rate of around 1.00, despite their limited impact on overall performance.

Regarding the choice of CUSUM parameters, we acknowledge that our explanation in Appendix D.2 was not sufficiently detailed and the selection criteria were missing. We have added these details to Appendix D.2 of the revised manuscript.

Question 6:

We have added evaluations of two types of adaptive adversaries with knowledge of the MATRIX defense mechanism in Section 5.4 of the revised submission: (1) Stealth, which specifically targets the detector in MATRIX, aiming to perturb agents' observations while evading detection; and (2) Dynamic, which adapts to the entire end-to-end defense pipeline of MATRIX, including its observation replacement mechanism. We trained Dynamic as an RL agent whose objective is to minimize the team reward in the presence of MATRIX's test-time defense. In particular, this adversary can potentially learn when perturbations are most effective, accounting for their subsequent effects on observation generation and CUSUM detection, and can thereby exploit the mismatch highlighted by the reviewer.

The results (Table 3 and Table 9 (Appendix E.3) in the revised submission) show that both types of adaptive adversaries can reduce the detection rate and increase the detection delay. However, their impact on overall team performance remains relatively limited, and MATRIX continues to outperform the baselines under attack. A lower detection rate and longer detection delay imply that more perturbed observations can evade detection and enter agents' observation histories. Nevertheless, the limited performance degradation suggests that the perturbations capable of evading detection generally have little influence on MATRIX's overall test-time operation. Conversely, perturbations that substantially disrupt the observation generation and decision-making processes are likely to be detected and mitigated by the defense mechanism (Please also refer to Figure 9 and the accompanying discussion in Appendix E.3 of the revised submission on this matter).

Many thanks for the reponse. I have raised my score.

Reviewer_jhZoOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MATRIX, a diffusion model-based MARL training framework that addresses the sample inefficiency of adversarial training.

Critical strengths
  1. The experimental results are comprehensive.

  2. The proposed method addresses the sample inefficiency of adversarial training.

Critical weaknesses

The generalizability of the proposed method requires further validation. Please refer to the questions section for details.

Questions for authors
  1. Is the proposed world model generalizable to other settings, particularly those where images are used as inputs?

  2. Are the world models used in WM-ATSA and WM-PAAD learned using the approach proposed in MATRIX? Why do WM-ATSA and WM-PAAD fail to outperform ATSA and PAAD in most cases? Does this suggest that the proposed world model approach is only effective within the MATRIX architecture?

  3. The authors should include additional robust learning baselines for comparison, such as PR [1] and ERNIE [2].

[1] Enhancing the robustness of qmix against state-adversarial attacks.

[2] Robust multiagent reinforcement learning via adversarial regularization: Theoretical foundation and stable algorithms.

1 comment

We thank the reviewer for their comments. Our responses are provided below. We have revised the manuscript accordingly, with the main changes in the text highlighted in blue.

Weakness1, Question1:

We have included an evaluation of MATRIX on the Commons Harvest scenario from the Melting Pot environment in Appendix E.4 of the revised submission. In this environment, the agents receive RGB frames as their observations. The training procedure and the test-time protocol of MATRIX are unchanged, and the main modifications are that the networks operating on observations become convolutional. Since most of our baselines do not support image-based observations, we limited the comparison to MAPPO and ERNIE. Figure 10(b) shows that MATRIX achieves higher returns with only 50K training samples, even when the baselines are given a budget of 200K samples. Furthermore, under FGSM attacks, MATRIX exhibits a performance degradation comparable to that of ERNIE (3.3% versus 3.7%), while the non-robust MAPPO baseline suffers a substantially larger performance drop of approximately 34%.

Question 2:

No, they were trained using one of the baseline world models. Since existing adversarial training approaches are model-free and sample-inefficient, we considered two ways of evaluating or adapting world-model-based approaches for adversarial robustness. First, we evaluated the robustness of policies trained using existing world models and compared them against MATRIX. The results, reported in Table 8 of Appendix E.1 in the revised submission, demonstrate that these policies exhibit poor robustness against adversarial attacks.

Second, we used the generative capabilities of one of the pretrained baseline world models (the one achieving better clean performance) to train existing model-free robustification methods. This is precisely what WM-ATSA and WM-PAAD represent. Since the real-environment interaction budget has already been consumed to train the world model, ATSA and PAAD must be trained exclusively on imagined trajectories generated by the pretrained model. The poor performance of WM-ATSA and WM-PAAD compared to their model-free counterparts trained in the real environment can be attributed to the distribution shift between trajectories experienced by the world model during training, and the imagined trajectories required by the respective robust method. We have investigated this effect in details in Appendix B. MATRIX, on the contrary, is an end-to-end framework, which has accurate and usable world-model training and test-time robustness together in one place.

Question 3:

We have added the evaluation of ERNIE and PR to Table 1 of the revised submission.

Reviewer_bK61OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MATRIX, a diffusion-based multi-agent reinforcement learning framework that integrates sample-efficient policy learning with adversarially robust decentralized execution. By selectively dropping global conditioning information during world-model training, MATRIX enables a shared diffusion model to support both globally conditioned imagination and locally conditioned observation prediction.

The framework further introduces policy robustification through mixed-mode imagined trajectories and a test-time defense mechanism that combines denoising-loss-based anomaly detection with CUSUM. When suspicious observations are detected, they are replaced with locally generated predictions. Experiments on SMAC and MaMuJoCo demonstrate improvements in sample efficiency and robustness against several observation attacks.

Critical strengths
  1. MATRIX integrates globally conditioned policy learning, locally conditioned imagination, and test-time observation correction within a shared world model. This is a meaningful system-level contribution that avoids training separate models for decentralized defense.
  2. Experiments across discrete and continuous control tasks demonstrate substantial improvements in sample efficiency and adversarial robustness. The ablations provide direct evidence that both test-time defense and policy robustification contribute to robustness.
Critical weaknesses
  1. The considered attacks do not explicitly optimize against the complete detection-and-replacement pipeline. An informed adversary could potentially manipulate observations while evading the denoising-loss-based detector, making the robustness claims insufficiently validated against defense-aware attacks.
  2. Several robust baselines exhibit substantially lower clean performance, making absolute attacked performance an imperfect measure of robustness. Moreover, detection quality and false-alarm behavior are not sufficiently quantified, and the main learning experiments use only three random seeds.
Questions for authors
  1. How does MATRIX perform against adaptive white-box attacks that jointly target policy performance and evasion of the CUSUM-based detector?

  2. Can the authors provide detection rates, false-positive rates, and detection delays across different attack budgets?

  3. How much of MATRIX's robustness advantage remains when comparing relative performance degradation against baselines with similar clean performance?

1 comment

We thank the reviewer for their comments. Our responses are provided below. We have revised the manuscript accordingly, with the main changes in the text highlighted in blue.

Weakness 1, Question 1:

We have added results for two types of adaptive attacks in Section 5.4 of the revised manuscript. The first attack, called Stealth, specifically targets the detector in MATRIX, aiming to perturb agents' observations while evading detection. We achieve this by incorporating a penalty term into the MAD attack objective that discourages large CUSUM scores. The other attack, called Dynamic, is a stronger adversary that adapts to the entire end-to-end defense pipeline of MATRIX, including its observation replacement mechanism. We formulate Dynamic as a reinforcement learning problem, where the adversary is trained to minimize the team reward in the presence of the defense mechanism.

The results in Table 3 of the revised manuscript show that although Stealth can substantially reduce the detection rate, its impact on team performance remains limited. For example, in 2s3z, Stealth reduces the detection rate from approximately 1.00 under non-adaptive attacks (Table 9) to 0.14, while causing only a marginal decrease in team performance (0.88 compared to 0.92 in the clean setting). This suggests that perturbations capable of evading detection typically have little influence on MATRIX's observation generation and decision-making processes. In contrast, Dynamic represents a more challenging attack scenario for MATRIX, as it explicitly accounts for the defense mechanism during optimization. As expected, Dynamic generally causes greater performance degradation than non-adaptive attacks. Nevertheless, the overall performance degradation remains relatively limited, demonstrating the effectiveness of MATRIX even against adaptive adversaries.

Weakness2 (on 3 random seeds):

We have conducted experiments with two additional random seeds and updated the learning curves (Figure 1) in the revised manuscript with results averaged over a total of five seeds. The overall results remain largely consistent with those previously reported. However, MATRIX now achieves slightly higher final average performance than MARIE in MMM and slightly lower performance than DIMA in CHC (although very close).

Question 2, Weakness 2 (on detection metrics):

We have included additional detection metrics in Table 3 for adaptive attacks, as well as for other attacks in Appendix E.3 of the revised submission (Table 9 and Figure 9).

Question3, Weakness2 (on comparison with similar clean performance baselines):

We had already compared the robustness of MATRIX against world-model-based approaches with comparable clean performance (selecting the best-performing runs of DIMA and MARIE) in Appendix E.1 of the original manuscript. However, we acknowledge that this comparison was not sufficiently detailed. To make the results clearer, we have updated Table 8 in Appendix E.1 to explicitly report the percentage performance degradation under attacks relative to clean performance, and added a more clear reference to this result in the main body. The results demonstrate that even when MATRIX and the baselines achieve comparable clean performance, MATRIX exhibits substantially better robustness against adversarial attacks (e.g., 4.2% drop versus 50% in MMM).

A primary motivation of our work is sample efficiency, making comparisons in low-sample regimes particularly relevant. Existing robust MARL approaches are model-free, and their high sample complexity limits their ability to learn effective policies with restricted training budgets, even in the absence of attacks, as Table 1 shows. Nevertheless, to provide a more comprehensive evaluation, we have added comparisons between MATRIX and these approaches trained with substantially larger sample budgets. Figure 8 in Appendix E.2 of the revised submission presents these results, including settings where the methods achieve comparable clean performance. The results show that MATRIX remains competitive in many cases, even when the model-free baselines are given significantly more training samples.

Comments on the paper

1 comment

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts or updates to your assessment. Thanks for your thoughtful reviews and discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.