- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper presents MATRIX, a framework built on a diffusion world model, aimed at sample-efficient multi-agent RL and at decentralized defense against observation attacks at test time. The key idea is that global conditioning is dropped selectively, thus a single model can be used three different ways: for centralized policy learning, for robustifying the policy against locally predicted observations, and for replacing observations at execution in a decentralized manner. Whenever the detector thinks an observation looks suspicious, replacement is triggered. The detector is a CUSUM test on the denoising loss. Experiments on SMAC and MaMuJoCo cover sample efficiency, adversarial robustness, and execution overhead.
Critical strengths
S1. Reusing one world model for both policy learning and decentralized observation defense is a sensible design, and the paper follows it through consistently. S2. The experiments cover both discrete and continuous action spaces, and the ablations give at least some support for the roles of policy robustification and test-time defense.
Critical weaknesses
W1. The proposed framework lacks novelty. Every component already exists, and the contribution is mainly in how they are wired together. And I could not tell what the robustification step buys you over simply training longer or training better. The paper never separates the two. W2. The experiments do not support the robustness claims as strongly as the paper suggests. The baselines are not matched in clean performance, the detector is not evaluated on its own terms, and the gap between what the policy sees during training and what it sees at execution is not tested.
Questions for authors
- I do not see much novelty beyond integration. The individual pieces are established techniques, and what is being claimed is essentially the architecture plus the training procedure.
- The world-model baselines fail, and the paper offers model distribution shift as the explanation. That may be right, but the experiments shown do not really establish it in every setting. It is asserted more than demonstrated.
- The detector assumes a harmful perturbation raises the locally conditioned denoising loss, yet no detection rate, no false-alarm rate, and no detection delay are reported. Appendix D.2 also chooses thresholds from FGSM and ATSA episodes, but never says what the numerical constraints on that choice were.
- Table 1 compares methods whose clean performance differs enormously. MATRIX reaches 5267 on Ant while the model-free robust baselines sit around 420–460. With starting points that far apart, a higher attacked return mainly shows better performance within the sample budget. It does not isolate greater resistance to perturbations.
- WM-ATSA and WM-PAAD are trained against frozen models that were learned under different policies, whereas MATRIX gets to alternate between policy learning and model updates. Their clean win rates are near zero, so the policies had not learned the task before robustness was even measured. These results do not isolate the advantage of MATRIX's robustification.
- During robustification the selected agents use their own predictions for the whole imagined rollout. At execution, however, corrupted observations the detector misses still enter the conditioning history, and agents flip between received and predicted observations according to CUSUM. This training–execution mismatch is never evaluated, even though the accuracy of later detection and replacement depends on it directly.
3 comments
Rebuttal (Part 2)
Question 2:
We have extended the analysis in Appendix B of the revised manuscript. Table 4 reports, for all six scenarios, the loss ratio between ATSA and the co-trained policy and the real-versus-simulated reward gap, with the metrics defined in the experimental setup. Figure 5 shows the MaMuJoCo reward comparison and complements the existing SMAC results (Figure 4). These results show that in each scenario, at least one measure is considerably different for ATSA compared to the co-trained policy. It is not straightforward to isolate the effect of the world model-policy mismatch, since the policies visit different trajectories. However, these results indicate that the world model generalizes poorly to unseen policies in all considered scenarios.
Question 3:
We have added the evaluation of MATRIX's detector performance in Table 3 (for newly-added adaptive attacks), and in more detail in Table 9 and Figure 9 in Appendix E.3 of the revised manuscript for other attacks. The results show that, for example, in 2s3z, MATRIX's detector can identify even small perturbations with a budget of 0.05 with a detection rate of around 1.00, despite their limited impact on overall performance.
Regarding the choice of CUSUM parameters, we acknowledge that our explanation in Appendix D.2 was not sufficiently detailed and the selection criteria were missing. We have added these details to Appendix D.2 of the revised manuscript.
Question 6:
We have added evaluations of two types of adaptive adversaries with knowledge of the MATRIX defense mechanism in Section 5.4 of the revised submission: (1) Stealth, which specifically targets the detector in MATRIX, aiming to perturb agents' observations while evading detection; and (2) Dynamic, which adapts to the entire end-to-end defense pipeline of MATRIX, including its observation replacement mechanism. We trained Dynamic as an RL agent whose objective is to minimize the team reward in the presence of MATRIX's test-time defense. In particular, this adversary can potentially learn when perturbations are most effective, accounting for their subsequent effects on observation generation and CUSUM detection, and can thereby exploit the mismatch highlighted by the reviewer.
The results (Table 3 and Table 9 (Appendix E.3) in the revised submission) show that both types of adaptive adversaries can reduce the detection rate and increase the detection delay. However, their impact on overall team performance remains relatively limited, and MATRIX continues to outperform the baselines under attack. A lower detection rate and longer detection delay imply that more perturbed observations can evade detection and enter agents' observation histories. Nevertheless, the limited performance degradation suggests that the perturbations capable of evading detection generally have little influence on MATRIX's overall test-time operation. Conversely, perturbations that substantially disrupt the observation generation and decision-making processes are likely to be detected and mitigated by the defense mechanism (Please also refer to Figure 9 and the accompanying discussion in Appendix E.3 of the revised submission on this matter).
Many thanks for the reponse. I have raised my score.
Rebuttal
We thank the reviewer for their comments. Our responses are provided below. We have revised the manuscript accordingly, with the main changes in the text highlighted in blue.
Weakness1, Question 1 (on novelty):
We agree that several building blocks, such as diffusion models and classifier-free guidance, are established. However, MATRIX contributes their adaptation and coordinated use for sample-efficient policy learning and decentralized robustness in MARL. Specifically, (i) we adapt classifier-free guidance to the CTDE information structure by dropping global conditioning while preserving each agent’s local context (local conditioning), allowing one shared world model to make both globally and locally conditioned predictions. This allows agents to perform imagination only based on their local information, a capability absent from existing world models; (ii) we introduce a mixed-mode imagination procedure that uses these local predictions to expose policies to prediction errors and their effects on coordination; and (iii) we propose a decentralized test-time defense that uses the model’s local denoising loss for anomaly detection and performs full diffusion sampling only when triggered. These methodological contributions together allow us to address an underexplored problem: sample-efficient learning of adversarial robustness in MARL.
Weakness 1, Question 4, Question 5 (on the gains of robustification & baselines with matched clean performance):
Regarding the benefits of MATRIX's robustification procedure compared to simply training for longer, we would first like to emphasize that the primary motivation of our work is sample efficiency. Accordingly, the most relevant comparisons are those conducted under a fixed, limited budget of interactions with the real environment. Under this constraint, existing model-free approaches struggle to learn effective policies, as demonstrated in Table 1.
This leaves two main alternatives for existing approaches. The first is to use a general-purpose world model, such as DIMA or MARIE, to learn a policy. These approaches can achieve competitive clean performance but lack robustness against test-time attacks. We had already compared the robustness of MATRIX against these methods, selecting the best-performing run of DIMA and MARIE, in Appendix E.1 of the original manuscript. However, we acknowledge that this comparison was not sufficiently detailed. To clarify the results, we have updated Table 8 in Appendix E.1 to explicitly report the percentage performance degradation under attacks relative to clean performance and added a clearer reference to this comparison in the main text. The results demonstrate that even when MATRIX and the baselines achieve comparable clean performance, MATRIX exhibits substantially greater robustness against adversarial attacks (e.g., a 4.2% performance drop for MATRIX versus 50% for MARIE in MMM against ATSA attack). Importantly, these approaches follow the same alternating model and policy update procedure and are trained under the same sample budget as MATRIX. Thus, the improved robustness can be attributed to the robustification-related mechanisms introduced by MATRIX during training and test-time.
The other possibility is to use the generative capability of a trained world-model and apply one of the model-free robustification approaches. Since the real sample budget has already been consumed to train an accurate world model, the training of the model-free approaches must rely exclusively on imagined trajectories generated by the frozen world model. Although these methods can use an unlimited number of imagined samples, the results for WM-ATSA and WM-PAAD in Table 1 show that this approach also falls short in terms of robustness.
Finally, to provide a more comprehensive evaluation, we have added comparisons between MATRIX and the model-free baselines trained with substantially larger sample budgets. Figure 8 in Appendix E.2 of the revised manuscript presents these results, including settings where the methods achieve comparable clean performance. The results show that, Compared to robust methods (ATSA, PPAD), MATRIX remains competitive in many cases, even when these model-free baselines are given significantly more training samples (e.g., see 2s3z in Figure 8). The figure also includes MAPPO, a non-robust baseline, which exhibits substantially greater performance degradation under adversarial attacks compared to robust methods (e.g., approximately 62.5% versus 9.8% for MATRIX in 2s3z) . This further confirms that longer training alone does not necessarily lead to improved robustness.
Thus, in summary, (1) Longer training alone does not guarantee robustness for algorithms without explicit robustification mechanisms, and (2) MATRIX achieves robustness comparable to that of existing robustification methods while requiring substantially fewer real-environment training samples.