- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies temporal objective aliasing in decentralized multi-agent reinforcement learning and proposes MODAL, which combines history encoding and weighted mean-field communication with a local diffusion world model. The model learns environment dynamics from online interaction data and generates imagined trajectories to train PPO policies. Experiments are conducted in the MOSMAC and Flatland environments.
Critical strengths
- Clear and well-structured framework combining history encoding, weighted mean-field communication, and local diffusion world modeling.
- Ablation studies on the history encoder and communication mechanism, together with theoretical analyses of trajectory distribution errors and value differences.
Critical weaknesses
- No supplementary code included
- Limited experimental coverage, mainly homogeneous agents and relatively small teams of up to 12 tanks or 20 trains.
Questions for authors
- To my knowledge, NetWorld[1] already combines diffusion, mean-field communication, and DTDE, but uses offline pretraining. Could the authors explain the main advances of this paper over such methods? What is the main novelty?
- [Lines 892–893] Each time, an -step trajectory is generated, but only the first step is retained, so the process still proceeds recursively. How does this design reduce error accumulation? In addition, each imagined step requires 16 DDIM sampling steps. What are the computational costs and resource requirements here?
- [Lines 423–431, 448–474] The existing ablations show that history and communication are useful, but do not distinguish whether the gains come from general state memory and additional information, or from specifically mitigating temporal objective aliasing. Stage clustering also does not directly show that the model can distinguish cases with similar current observations but different histories and action values.
- Why choose DTDE rather than CTDE? Diffusion itself does not rule out centralized training. Could the authors clarify this point?
If the authors address the above questions, I will increase my score.
References
[1] NetWorld: Communication-Based Diffusion World Model for Multi-Agent Reinforcement Learning in Wireless Networks.
8 comments
Responses to Questions
Q1. Novelty and advances over NetWorld.
We appreciate the reviewer for highlighting NetWorld. Although both methods employ diffusion world models and mean-field communication within the DTDE paradigm, they differ fundamentally in their research objectives, communication mechanisms, and use of world models.
First, NetWorld primarily focuses on cross-task generalization, leveraging offline pretraining of a shared diffusion world model, followed by target-task adaptation and training of task-specific interface models. In contrast, MODAL addresses temporal objective aliasing within a task, where similar local observations may correspond to distinct task stages and continuation-value profiles. To address this challenge, MODAL integrates recurrent history representations with local diffusion world models to capture history-dependent, multimodal future dynamics. Furthermore, unlike the simple mean-field aggregation adopted in NetWorld, MODAL introduces a distance-weighted mean-field mechanism that incorporates both spatial proximity and neighborhood density information. This allows agents to better characterize heterogeneous neighborhood interactions and adapt their decisions to dynamically evolving task objectives.
Second, the two methods employ diffusion world models differently. NetWorld uses diffusion models to predict future local observation trajectories and subsequently derives actions through an inverse dynamics model, requiring iterative diffusion sampling and inverse dynamics inference during execution. In contrast, MODAL integrates diffusion-based imagination with actor-critic learning, using generated trajectories to enrich policy training and capture long-term consequences of actions. By leveraging diffusion-generated trajectories for actor-critic optimization, MODAL incorporates predictive information into policy learning and enables direct action selection without online diffusion sampling. This decoupling of world-model imagination from online action selection substantially reduces execution complexity while retaining the advantages of multi-step predictive learning.
Therefore, MODAL provides a history-conditioned diffusion world-model framework specifically designed to mitigate temporal objective aliasing, with density-aware neighborhood coordination and efficient decentralized policy execution.
Q2. Error accumulation and computational efficiency.
MODAL adopts a receding-horizon mechanism that generates an -step trajectory but retains only the first predicted transition to advance the imagined rollout. This allows the world model to capture multi-step temporal dependencies while avoiding direct reliance on distant open-loop predictions. Specifically, at each imagination step, the predicted observation and reward are incorporated into the updated history, and a new -step trajectory is generated conditioned on the updated history and action. By repeatedly regenerating predictions rather than directly following an entire previously predicted trajectory, this mechanism mitigates the direct propagation of distant prediction errors.
Importantly, Table 1 quantitatively evaluates error accumulation under this recursive imagination mechanism. As detailed in Appendix E.2, all methods start from the same ground-truth histories and use identical ground-truth action sequences. Predictions are recursively fed back into subsequent steps without teacher forcing, and cumulative observation prediction errors are measured over 4- and 8-step rollouts. MODAL consistently achieves the lowest errors at both horizons, demonstrating more accurate recursive trajectory prediction and reduced accumulated prediction errors compared with the evaluated baselines. Furthermore, Theorem 1 provides an upper bound on the distributional approximation error of a single -step trajectory prediction under the stated assumptions, complementing the empirical evaluation of recursive prediction accuracy.
Regarding computational efficiency, as clarified in Q1, diffusion sampling is required only during training, where imagined trajectories facilitate policy optimization. During execution, each agent directly invokes its learned policy without iterative diffusion denoising, avoiding the additional sampling overhead in online action selection.
To further quantify this training-deployment trade-off, we conduct additional computational efficiency experiments on MOSMAC 3t and 4t, using a single NVIDIA RTX 4090 GPU across all methods. Beyond the baselines included in our original evaluation, we additionally introduce two recent model-based MARL methods, MARIE [1] and DMAWM [2], to provide a more comprehensive comparison. As reported in Table R-2, MODAL requires considerably less training time than MAMBA and MARIE. Moreover, its deployment parameter count is substantially smaller than those of MAMBA, MARIE, and DMAWM, while remaining comparable to DIMA. MODAL also achieves lower inference latency than MAMBA, MARIE, and DMAWM, with considerably lower peak inference memory than MAMBA and DMAWM. These results highlight the practical advantage of MODAL in leveraging diffusion-based imagination for policy optimization while maintaining lightweight decentralized execution. We further incorporate these computational efficiency comparisons into the newly added Section 6.6 Computational Efficiency of the revised manuscript.
Table R-2. Computational efficiency comparison on MOSMAC 3t and 4t.
| Method | Train Time (h) | Deploy Params (M) | Latency (ms) | Peak Inf. Mem. (MiB) |
|---|---|---|---|---|
MOSMAC 3t | ||||
| MODAL | 8.28 | 0.038 | 1.031 | 16.39 |
| IPPO | 3.64 | 0.028 | 0.308 | 9.25 |
| MAPPO | 5.07 | 0.007 | 0.307 | 9.17 |
| IA2C | 2.98 | 0.028 | 0.348 | 9.25 |
| MAMBA | 83.45 | 11.626 | 2.564 | 68.42 |
| MARIE | 34.22 | 1.341 | 1.267 | 14.80 |
| DMAWM | 7.51 | 37.345 | 2.346 | 157.22 |
| DIMA | 5.06 | 0.039 | 0.699 | 9.28 |
MOSMAC 4t | ||||
| MODAL | 8.36 | 0.040 | 0.892 | 16.44 |
| IPPO | 3.82 | 0.029 | 0.353 | 9.25 |
| MAPPO | 5.35 | 0.008 | 0.339 | 9.18 |
| IA2C | 2.83 | 0.029 | 0.329 | 9.25 |
| MAMBA | 70.57 | 11.635 | 2.475 | 68.50 |
| MARIE | 34.44 | 1.367 | 1.215 | 14.92 |
| DMAWM | 6.99 | 37.368 | 1.847 | 157.36 |
| DIMA | 4.42 | 0.040 | 0.727 | 9.28 |
References:
[1] Y. Zhang, C. Bai, B. Zhao, J. Yan, X. Li, and X. Li, “Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models,” Transactions on Machine Learning Research (TMLR), 2025.
[2] D. Xue, J. Jiang, S. Zhang, W. Guo, L. Yuan, Z. Zhang, and Y. Yu, “Learning Disentangled Multi-Agent World Model for Decentralized Control,” in Proceedings of the 43rd International Conference on Machine Learning (ICML), pp. 143194–143215, 2026.
We sincerely thank the reviewer for the constructive comments and recognition of our framework and experimental results. We address the concerns and questions as follows.
Response to Critical Weaknesses
W1. Code availability.
We have made our implementation publicly available through the link provided in the Reproducibility Statement to facilitate reproducibility and further research.
W2. Limited experimental coverage.
We appreciate the reviewer’s concern regarding experimental coverage. To further evaluate scalability and generalizability, we conduct additional zero-shot team-size transfer experiments on Flatland and include the results in the newly added Section 6.5 Zero-Shot Team-Size Generalization. Specifically, fully trained models using 20 agents are evaluated on teams of 20, 40, 60, 80, and 100 agents, with all learned parameters frozen and no retraining or fine-tuning. We maintain a fixed 40×60 map with 6 cities and unchanged environment configurations, thereby examining generalization to increasing team sizes and traffic densities under fixed railway capacity. For MAMBA, only the non-learnable sinusoidal positional encodings are extended to accommodate larger teams. All methods are evaluated over 8 training seeds, with 20 episodes per seed.
Table R-1. Zero-shot team-size transfer on Flatland. Test success rates (%) over 20 evaluation episodes per seed, reported as mean and sample standard deviation across 8 training seeds.
As shown in Table R-1, MODAL achieves the highest mean test success rate across all team sizes. At 40 agents, MODAL reaches 11.36, nearly doubling the strongest baseline (6.17). Even when directly transferred to 100 agents, MODAL maintains the highest mean test success rate (2.94), compared with 2.48 for MAPPO. Although all methods experience substantial performance degradation under increasingly congested railway conditions, these results demonstrate MODAL's comparative advantage in zero-shot transfer to unseen team sizes without additional training.