acceptodds
Under review as a conference paper at ICLR 2027

Diffusion World Models for Temporal Objective Aliasing in Decentralized MARL

Abstract

World models for multi-agent reinforcement learning (MARL) face a fundamental ambiguity under partial observability: the same local observation can correspond to multiple long-horizon objectives, inducing distinct future trajectory distributions. We refer to this phenomenon as *temporal objective aliasing*. Unlike conventional perceptual aliasing, which arises from ambiguity in the underlying state, temporal objective aliasing arises when historical interaction contexts imply different objectives despite similar current observations. Consequently, a world model conditioned only on local observations may collapse multiple future modes, leading to inaccurate imagined trajectories and suboptimal policy updates. Moreover, centralized multi-agent world models face combinatorial growth in joint state-action spaces, whereas purely local models may overlook coordination-relevant interactions and yield suboptimal decisions. To address these challenges, we propose **MODAL** (**M**odel-based **O**bjective-aware **D**iffusion for M**A**R**L**), a decentralized framework where each agent models local long-horizon trajectories with a diffusion world model and summarizes neighborhood interactions through weighted mean field (MF) communication. A history encoder further infers latent objectives from local histories to guide trajectory prediction and policy optimization. We establish theoretical guarantees on MODAL's ability to preserve continuation-value distinctions under temporal objective aliasing. Experiments on MOSMAC and Flatland show that MODAL outperforms established MARL baselines, with larger gains as team size and coordination complexity grow.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Diffusion World Models, Temporal Objective Aliasing, Decentralized Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kA6eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies temporal objective aliasing in decentralized multi-agent reinforcement learning and proposes MODAL, which combines history encoding and weighted mean-field communication with a local diffusion world model. The model learns environment dynamics from online interaction data and generates imagined trajectories to train PPO policies. Experiments are conducted in the MOSMAC and Flatland environments.

Critical strengths
  1. Clear and well-structured framework combining history encoding, weighted mean-field communication, and local diffusion world modeling.
  2. Ablation studies on the history encoder and communication mechanism, together with theoretical analyses of trajectory distribution errors and value differences.
Critical weaknesses
  1. No supplementary code included
  2. Limited experimental coverage, mainly homogeneous agents and relatively small teams of up to 12 tanks or 20 trains.
Questions for authors
  1. To my knowledge, NetWorld[1] already combines diffusion, mean-field communication, and DTDE, but uses offline pretraining. Could the authors explain the main advances of this paper over such methods? What is the main novelty?
  2. [Lines 892–893] Each time, an HH-step trajectory is generated, but only the first step is retained, so the process still proceeds recursively. How does this design reduce error accumulation? In addition, each imagined step requires 16 DDIM sampling steps. What are the computational costs and resource requirements here?
  3. [Lines 423–431, 448–474] The existing ablations show that history and communication are useful, but do not distinguish whether the gains come from general state memory and additional information, or from specifically mitigating temporal objective aliasing. Stage clustering also does not directly show that the model can distinguish cases with similar current observations but different histories and action values.
  4. Why choose DTDE rather than CTDE? Diffusion itself does not rule out centralized training. Could the authors clarify this point?

If the authors address the above questions, I will increase my score.

References

[1] NetWorld: Communication-Based Diffusion World Model for Multi-Agent Reinforcement Learning in Wireless Networks.

8 comments

We sincerely thank the reviewer for the constructive comments and recognition of our framework and experimental results. We address the concerns and questions as follows.

Response to Critical Weaknesses

W1. Code availability.

We have made our implementation publicly available through the link provided in the Reproducibility Statement to facilitate reproducibility and further research.

W2. Limited experimental coverage.

We appreciate the reviewer’s concern regarding experimental coverage. To further evaluate scalability and generalizability, we conduct additional zero-shot team-size transfer experiments on Flatland and include the results in the newly added Section 6.5 Zero-Shot Team-Size Generalization. Specifically, fully trained models using 20 agents are evaluated on teams of 20, 40, 60, 80, and 100 agents, with all learned parameters frozen and no retraining or fine-tuning. We maintain a fixed 40×6040\times60 map with 6 cities and unchanged environment configurations, thereby examining generalization to increasing team sizes and traffic densities under fixed railway capacity. For MAMBA, only the non-learnable sinusoidal positional encodings are extended to accommodate larger teams. All methods are evaluated over 8 training seeds, with 20 episodes per seed.

Table R-1. Zero-shot team-size transfer on Flatland. Test success rates (%) over 20 evaluation episodes per seed, reported as mean and sample standard deviation across 8 training seeds.

AgentsMODALIPPOMAPPOIA2CMAMBA
2032.46±14.49\mathbf{32.46}_{\pm14.49}14.47±2.6914.47_{\pm2.69}11.88±4.1611.88_{\pm4.16}12.97±9.4412.97_{\pm9.44}11.22±7.9511.22_{\pm7.95}
4011.36±3.87\mathbf{11.36}_{\pm3.87}5.83±1.155.83_{\pm1.15}6.17±1.216.17_{\pm1.21}5.47±3.245.47_{\pm3.24}5.80±3.045.80_{\pm3.04}
606.74±4.12\mathbf{6.74}_{\pm4.12}3.70±0.513.70_{\pm0.51}3.76±0.653.76_{\pm0.65}2.82±1.792.82_{\pm1.79}3.94±1.913.94_{\pm1.91}
804.65±4.00\mathbf{4.65}_{\pm4.00}2.57±0.552.57_{\pm0.55}2.84±0.412.84_{\pm0.41}1.89±1.061.89_{\pm1.06}2.59±1.392.59_{\pm1.39}
1002.94±1.42\mathbf{2.94}_{\pm1.42}2.12±0.352.12_{\pm0.35}2.48±0.352.48_{\pm0.35}1.42±0.601.42_{\pm0.60}2.42±1.492.42_{\pm1.49}

As shown in Table R-1, MODAL achieves the highest mean test success rate across all team sizes. At 40 agents, MODAL reaches 11.3611.36\\%, nearly doubling the strongest baseline (6.176.17\\%). Even when directly transferred to 100 agents, MODAL maintains the highest mean test success rate (2.942.94\\%), compared with 2.482.48\\% for MAPPO. Although all methods experience substantial performance degradation under increasingly congested railway conditions, these results demonstrate MODAL's comparative advantage in zero-shot transfer to unseen team sizes without additional training.

Responses to Questions

Q1. Novelty and advances over NetWorld.

We appreciate the reviewer for highlighting NetWorld. Although both methods employ diffusion world models and mean-field communication within the DTDE paradigm, they differ fundamentally in their research objectives, communication mechanisms, and use of world models.

First, NetWorld primarily focuses on cross-task generalization, leveraging offline pretraining of a shared diffusion world model, followed by target-task adaptation and training of task-specific interface models. In contrast, MODAL addresses temporal objective aliasing within a task, where similar local observations may correspond to distinct task stages and continuation-value profiles. To address this challenge, MODAL integrates recurrent history representations with local diffusion world models to capture history-dependent, multimodal future dynamics. Furthermore, unlike the simple mean-field aggregation adopted in NetWorld, MODAL introduces a distance-weighted mean-field mechanism that incorporates both spatial proximity and neighborhood density information. This allows agents to better characterize heterogeneous neighborhood interactions and adapt their decisions to dynamically evolving task objectives.

Second, the two methods employ diffusion world models differently. NetWorld uses diffusion models to predict future local observation trajectories and subsequently derives actions through an inverse dynamics model, requiring iterative diffusion sampling and inverse dynamics inference during execution. In contrast, MODAL integrates diffusion-based imagination with actor-critic learning, using generated trajectories to enrich policy training and capture long-term consequences of actions. By leveraging diffusion-generated trajectories for actor-critic optimization, MODAL incorporates predictive information into policy learning and enables direct action selection without online diffusion sampling. This decoupling of world-model imagination from online action selection substantially reduces execution complexity while retaining the advantages of multi-step predictive learning.

Therefore, MODAL provides a history-conditioned diffusion world-model framework specifically designed to mitigate temporal objective aliasing, with density-aware neighborhood coordination and efficient decentralized policy execution.

Q2. Error accumulation and computational efficiency.

MODAL adopts a receding-horizon mechanism that generates an HH-step trajectory but retains only the first predicted transition to advance the imagined rollout. This allows the world model to capture multi-step temporal dependencies while avoiding direct reliance on distant open-loop predictions. Specifically, at each imagination step, the predicted observation and reward are incorporated into the updated history, and a new HH-step trajectory is generated conditioned on the updated history and action. By repeatedly regenerating predictions rather than directly following an entire previously predicted trajectory, this mechanism mitigates the direct propagation of distant prediction errors.

Importantly, Table 1 quantitatively evaluates error accumulation under this recursive imagination mechanism. As detailed in Appendix E.2, all methods start from the same ground-truth histories and use identical ground-truth action sequences. Predictions are recursively fed back into subsequent steps without teacher forcing, and cumulative ℓ1\ell_1 observation prediction errors are measured over 4- and 8-step rollouts. MODAL consistently achieves the lowest errors at both horizons, demonstrating more accurate recursive trajectory prediction and reduced accumulated prediction errors compared with the evaluated baselines. Furthermore, Theorem 1 provides an upper bound on the distributional approximation error of a single HH-step trajectory prediction under the stated assumptions, complementing the empirical evaluation of recursive prediction accuracy.

Regarding computational efficiency, as clarified in Q1, diffusion sampling is required only during training, where imagined trajectories facilitate policy optimization. During execution, each agent directly invokes its learned policy without iterative diffusion denoising, avoiding the additional sampling overhead in online action selection.

To further quantify this training-deployment trade-off, we conduct additional computational efficiency experiments on MOSMAC 3t and 4t, using a single NVIDIA RTX 4090 GPU across all methods. Beyond the baselines included in our original evaluation, we additionally introduce two recent model-based MARL methods, MARIE [1] and DMAWM [2], to provide a more comprehensive comparison. As reported in Table R-2, MODAL requires considerably less training time than MAMBA and MARIE. Moreover, its deployment parameter count is substantially smaller than those of MAMBA, MARIE, and DMAWM, while remaining comparable to DIMA. MODAL also achieves lower inference latency than MAMBA, MARIE, and DMAWM, with considerably lower peak inference memory than MAMBA and DMAWM. These results highlight the practical advantage of MODAL in leveraging diffusion-based imagination for policy optimization while maintaining lightweight decentralized execution. We further incorporate these computational efficiency comparisons into the newly added Section 6.6 Computational Efficiency of the revised manuscript.

Table R-2. Computational efficiency comparison on MOSMAC 3t and 4t.

MethodTrain Time (h)Deploy Params (M)Latency (ms)Peak Inf. Mem. (MiB)
MOSMAC 3t
MODAL8.280.0381.03116.39
IPPO3.640.0280.3089.25
MAPPO5.070.0070.3079.17
IA2C2.980.0280.3489.25
MAMBA83.4511.6262.56468.42
MARIE34.221.3411.26714.80
DMAWM7.5137.3452.346157.22
DIMA5.060.0390.6999.28
MOSMAC 4t
MODAL8.360.0400.89216.44
IPPO3.820.0290.3539.25
MAPPO5.350.0080.3399.18
IA2C2.830.0290.3299.25
MAMBA70.5711.6352.47568.50
MARIE34.441.3671.21514.92
DMAWM6.9937.3681.847157.36
DIMA4.420.0400.7279.28

References:

[1] Y. Zhang, C. Bai, B. Zhao, J. Yan, X. Li, and X. Li, “Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models,” Transactions on Machine Learning Research (TMLR), 2025.

[2] D. Xue, J. Jiang, S. Zhang, W. Guo, L. Yuan, Z. Zhang, and Y. Yu, “Learning Disentangled Multi-Agent World Model for Decentralized Control,” in Proceedings of the 43rd International Conference on Machine Learning (ICML), pp. 143194–143215, 2026.

Reviewer_5DKHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies temporal objective aliasing in decentralized MARL, where similar current observations can mask different history-dependent decision contexts. MODAL combines a learned history representation, mean-field aggregation, and a diffusion world model to generate imagined rollouts for policy training. The learned policy executes without diffusion sampling. Experiments on MOSMAC and Flatland report improvements over the evaluated baselines, particularly in several harder settings.

Critical strengths
  1. The paper addresses a practical challenge: using history to resolve ambiguity in decentralized multi-agent decision-making.

  2. The reported results are promising on harder MOSMAC scenarios and larger Flatland settings, with evaluation across eight training seeds.

  3. History and mean-field ablations, together with prediction-quality comparisons, provide useful evidence about individual components.

Critical weaknesses
  1. In the submitted code, AgentAuxHeads (src/world_models/agent_diffusion.py) appears to produce identical predictions across future steps despite step-specific targets. This raises concerns about the auxiliary predictions used in imagined rollouts.

  2. Eq. (7) aggregates neighbors' own observations, but src/utils/mean_field.py pools features within each agent's observation. This discrepancy leaves the experimental information setting and baseline comparability unclear.

  3. The novelty beyond standard history-conditioned model-based MARL remains unclear. Non-diffusion alternatives are evaluated only for prediction accuracy, and MAMBA is the only model-based policy baseline, limiting attribution of the gains.

Questions for authors
  1. Does the submitted auxiliary-head implementation match the experiments? How does it distinguish future steps and reconcile multi-step supervision with first-step use?

  2. Which mean-field implementation was used, and were communication access and observation/reward histories matched across baselines?

  3. Would the gains persist with a standard history encoder or non-diffusion world model under matched inputs, capacity, and training budgets?

  4. Could the authors compare policy performance with recent model-based MARL methods such as MARIE or DMAWM, or explain why these comparisons are unsuitable?

5 comments

We sincerely thank the reviewer for the careful assessment and constructive comments. We appreciate the concerns regarding implementation consistency, experimental fairness, and methodological novelty, and provide the following clarifications.

Q1. Auxiliary prediction heads and temporal supervision.

We thank the reviewer for the careful examination of our implementation. We confirm that the submitted auxiliary-head implementation is consistent with our experiments. To prevent future-information leakage, the auxiliary heads use a shared conditioning vector derived solely from the current interaction context, without accessing ground-truth future observations or actions. This provides a simple and consistent conditioning mechanism for parallel multi-step supervision.

Specifically, the auxiliary heads are jointly supervised by step-specific targets over the HH-step horizon. Since their outputs are shared across future steps, they learn an aggregated future-window target rather than independent step-wise predictions. For binary targets, minimizing the summed BCE losses is equivalent to supervision by the average target across the prediction horizon. Thus, these auxiliary heads provide aggregated future information about rewards, termination, and action availability, complementing the diffusion model's multi-step observation predictions.

Importantly, MODAL adopts receding-horizon imagination: only the first predicted transition is retained, and subsequent predictions are regenerated from the updated imagined history. Accordingly, only the first auxiliary output is used to construct each imagined transition, while the remaining repeated outputs are discarded, avoiding temporal indexing mismatches from later-step outputs. As these heads serve as auxiliary components for constructing complete imagined trajectories, the current implementation primarily demonstrates the feasibility of the overall MODAL framework, whose effectiveness is supported by our experimental results. A promising extension is to incorporate prediction-step embeddings into the shared current-context conditioning, enabling step-specific auxiliary predictions without accessing ground-truth future information.

Q2. Mean-field implementation and baseline fairness.

We appreciate the reviewer for pointing out the apparent discrepancy between Eq. (7) and the implementation. This difference arises from the benchmark-specific design of local observations, as explained below.

Our formulation assumes decentralized decision-making based on locally available information. In both MOSMAC and Flatland, each agent's local observation already contains information related to neighboring agents. Therefore, rather than explicitly accessing and aggregating additional observations from other agents, our implementation constructs the mean-field representation from the neighborhood-related features already included in each agent's local observation. Eq. (7) describes the intended neighborhood aggregation at a conceptual level, while the implementation operates on the available local observation features.

Importantly, this implementation does not expand the original observation scope or introduce additional information beyond what is available in the benchmark-defined local observations. MODAL and all compared baselines make decisions based on the same locally available information, without granting MODAL any additional observation access or communication privileges. The mean-field module only reorganizes existing neighborhood-related features to facilitate decentralized decision-making, ensuring that performance differences arise from how the available information is utilized rather than disparities in information accessibility. Furthermore, as stated in the second paragraph of Section 6.1, we ensure a fair comparison among world-model-based methods by adopting identical historical context lengths and imagined rollout horizons for world-model prediction and simulated trajectory generation: 8 history steps and 4 rollout steps in MOSMAC, and 16 history steps and 8 rollout steps in Flatland. These consistent settings ensure that all world-model-based methods generate imagined trajectories under the same temporal context and rollout-horizon constraints, without providing MODAL with additional information or advantages in trajectory generation.

Q3. Contributions beyond standard history encoding and non-diffusion world models.

We thank the reviewer for this valuable suggestion. To further clarify the contributions of MODAL, we conduct additional experiments in the newly added Appendix E.3 Controlled Temporal Objective Aliasing Evaluation. Notably, MODAL already employs a standard GRU history encoder. We compare it with History-PPO+MF, which retains the same historical inputs and MF information but removes world-model imagination, and capacity-matched Gaussian WM, which replaces the diffusion predictor with a Gaussian model under comparable inputs and model capacity.

As shown in Fig. 11, MODAL achieves faster convergence on MOSMAC 8t and substantially stronger performance on the challenging 12t scenario, particularly in test win rate. These results indicate that MODAL's advantages extend beyond conventional recurrent memory and Gaussian world modeling, supporting the effectiveness of history-conditioned diffusion modeling in capturing objective-relevant temporal information and improving cooperative policy learning.

Q4. Comparisons with recent model-based MARL methods.

We thank the reviewer for this valuable suggestion. We have added comparisons with MARIE and DMAWM on MOSMAC 3t and 4t in Appendix E.1. Both methods follow their official implementations and hyperparameters, with only the update frequency, world-model history context length, and local observation trajectory horizon adjusted for fair comparison. Given the limited rebuttal period, we evaluate all methods over 5×1065\times10^6 environment steps using eight random seeds.

As shown in Fig. 8 of the revised manuscript, MODAL converges to high average test win rates and episode returns in both scenarios, whereas MARIE and DMAWM exhibit near-zero win rates with occasional minor fluctuations and substantially lower returns throughout the evaluated training period. Notably, these differences emerge even in the relatively simple, small-team MOSMAC 3t and 4t scenarios, suggesting potential difficulties for these baselines in capturing long-horizon dynamics and adapting to evolving objectives. Furthermore, the newly added Appendix E.5 Additional Computational Efficiency Evaluation shows that MARIE requires considerably longer training time, while both methods have substantially larger deployment models than MODAL. Together, these results support MODAL's effectiveness in handling complex temporal dynamics and shifting task objectives while maintaining efficient decentralized execution.

Thanks for clarifying the auxiliary heads. I have one follow-up about the initial action mask in imagined rollouts. agent_sequence_dataset.py:108–110 swaps the agent and action axes before flattening the mask, but modal_learner.py:188 reshapes it without swapping them back. Am I reading this correctly that this can mask out a valid action in MOSMAC? If so, why is the transpose needed?

Reviewer_x2ioOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MODAL, a decentralized diffusion world-model framework for cooperative multi-agent reinforcement learning (MARL). The authors introduce the concept of temporal objective aliasing, in which similar local observations arising from different interaction histories may imply different continuation-value profiles and action preferences. MODAL addresses this issue through three main components: a recurrent history encoder that infers objective-relevant latent representations, a diffusion model that predicts multi-step local observation trajectories, and distance-weighted mean-field (MF) communication that summarizes neighborhood interactions using fixed-dimensional messages. Auxiliary prediction heads estimate utility, termination, and available-action masks, and imagined rollouts are used for actor-critic policy optimization. The paper provides a theoretical analysis relating history-representation and mean-field approximation errors to trajectory-distribution and continuation-value errors. Experiments on MOSMAC and Flatland compare MODAL with model-free MARL baselines and the world-model-based methods MAMBA and DIMA. The reported results indicate improved policy learning in several challenging scenarios and lower multi-step observation prediction errors than the evaluated world-model baselines. Ablation studies examine the contributions of the history encoder, diffusion trajectory modeling, and weighted MF communication. The main questions concern the strength of the assumptions underlying the theoretical guarantees, consistency between independently modeled local trajectories, and the computational and empirical evidence supporting the claimed scalability.

Critical strengths
  1. A coherent framework for history-conditioned multimodal world modeling. The paper addresses a meaningful problem in partially observable MARL: current local observations may not adequately identify the task stage or the relevant future behavior. Conditioning a multi-step diffusion world model on recurrent history representations is a plausible way to preserve multiple future modes. The paper also includes direct ablations comparing the diffusion model with a Gaussian MLP and a one-step prediction model, which provide useful evidence for the proposed trajectory-modeling design.

  2. A reasonably broad experimental evaluation with relevant ablations. The authors evaluate MODAL on MOSMAC and Flatland, report results over eight random seeds, and examine policy performance, multi-step prediction error, history encoding, and weighted MF communication. For example, on MOSMAC 8t, the history-encoder ablation reduces the reported win rate from 97.44% to 59.24%, although the ablated result also has high variance. The multi-step prediction experiments further show consistent improvements over the evaluated alternatives. These results provide meaningful empirical support for the overall framework, even though the individual contributions and generality of the claims require further validation.

Critical weaknesses
  1. The theoretical guarantees rely on restrictive assumptions that substantially limit their practical interpretation. In Appendix B.1, the weighted MF approximation analysis assumes that the oracle conditional noise predictor decomposes as a weighted sum of per-neighbor predictions (Eq. 22). This is a strong structural assumption on the predictor, rather than a consequence of distance-weighted aggregation alone. The subsequent error bound therefore does not establish that an arbitrary nonlinear diffusion world model will be well approximated by the proposed MF representation. The analysis also assumes a uniformly bounded history-representation error and smoothness of the conditional predictor, while neglecting network fitting error. In addition, the paper acknowledges an endpoint singularity in the diffusion weighting term and relies on early stopping. These qualifications make the theoretical claims considerably more conditional than the abstract may suggest. The authors should clarify the scope of the guarantees and discuss how the assumptions could be assessed or justified in practice.

  2. Consistency between local world models and joint multi-agent dynamics remains insufficiently established. MODAL predicts each agent's future local observation trajectory using its own action, history representation, and compressed neighborhood message. However, in a cooperative environment, an agent's next observation is coupled to other agents' actions and states. Independently generated local trajectories may therefore be mutually inconsistent or fail to correspond to any realizable joint environment trajectory. Although weighted MF communication captures neighborhood information, the paper does not sufficiently establish that it preserves the cross-agent dependencies needed for reliable imagined policy optimization. The trajectory-distribution guarantee in Theorem 1 concerns a local observation trajectory under stated assumptions; it should not be interpreted as a guarantee of joint multi-agent trajectory consistency. This distinction matters because the imagined rollouts directly affect actor and critic updates.

  3. The scalability and practical efficiency claims are not fully supported by computational evidence. The experiments cover teams of up to 20 agents in Flatland and increasingly challenging MOSMAC scenarios, but the paper does not provide a sufficiently detailed comparison of wall-clock training time, inference latency, memory consumption, or computational cost per environment step. MODAL uses a diffusion model with 64 training diffusion timesteps and 16 DDIM sampling steps, together with recurrent history encoding, auxiliary heads, and imagined rollouts. Its sample-efficiency gains are promising, but environment steps alone do not measure the total computational budget. Moreover, the Flatland 20-agent success rate remains relatively low at 32.46% ± 14.49%, indicating that substantial coordination challenges remain. Stronger runtime measurements and larger-scale evaluations are needed to substantiate broad claims about scalability.

  4. The empirical evidence for temporal objective aliasing is suggestive but not fully diagnostic. The paper defines temporal objective aliasing through similar observations and different continuation-value profiles. The t-SNE visualization in Figure 5 shows stage-dependent structure in the learned history representations, and the history-encoder ablation demonstrates that history is useful in several difficult scenarios. However, these findings do not establish that the learned latent representation specifically resolves objective aliasing rather than encoding other correlated information, such as task progress, recent actions, or neighborhood dynamics. The current evaluation would be stronger with controlled pairs of histories that have similar current observations but different continuation-value profiles, together with quantitative measurements of whether the inferred representation distinguishes their action preferences. The paper should also clarify whether the claimed effect is distinct from conventional recurrent memory in partially observable MARL.

Questions for authors
  1. Theoretical assumptions: In Appendix B.1, Eq. 22 assumes that the oracle conditional noise predictor decomposes into a weighted sum of per-neighbor predictions. Under what conditions is this assumption reasonable for a nonlinear diffusion model? Can the authors provide empirical evidence validating this approximation, or clarify how the theoretical guarantees change when the assumption is relaxed?

  2. Joint trajectory consistency: During imagined rollouts, each agent predicts its own future local observations conditioned on local information and a weighted MF message. How do the authors ensure that the generated trajectories remain consistent with the coupled dynamics of the joint multi-agent environment? Have joint transition consistency or cross-agent prediction errors been measured?

  3. Objective aliasing validation: Can the authors construct controlled pairs of histories with similar current observations but substantially different continuation-value profiles, and quantitatively evaluate whether the learned history representations distinguish these cases? How does MODAL compare with a recurrent model-free baseline using an equivalent history encoder but no diffusion world model?

  4. Computational efficiency: What are the wall-clock training costs, inference latency, and memory requirements of MODAL compared with IPPO, MAPPO, MAMBA, and DIMA under comparable hardware and environment-interaction budgets? Can the authors provide evidence that the gains in sample efficiency translate into improved computational efficiency?

  5. Generalization and robustness: How well does MODAL generalize to unseen environment configurations, different team sizes, or altered neighborhood densities without extensive retraining? In particular, can the authors evaluate performance beyond the 20-agent Flatland setting and investigate the high variance observed in the more challenging scenarios?

5 comments

We sincerely thank the reviewer for the thorough evaluation and constructive feedback, as well as the recognition of our framework and experimental results. We appreciate the concerns regarding theoretical assumptions, joint trajectory consistency, computational efficiency, and empirical validation. We provide the following clarifications and additional analyses to address these concerns.

Q1. Theoretical assumptions of weighted mean-field approximation.

We thank the reviewer for this insightful question. The decomposition assumption underlying Eq. (22) is motivated by established mean-field MARL formulations [1, 2], which approximate multi-agent value functions through averaged or weighted pairwise interactions among neighboring agents.

Following this principle, we extend the pairwise interaction decomposition from value functions to conditional noise predictors in diffusion world models. Importantly, this assumption does not require the noise predictor to be linear, as each neighbor-dependent component can be nonlinear. Instead, it assumes that the dominant effects of neighboring agents can be represented through weighted pairwise contributions, which is reasonable in locally coupled multi-agent environments where higher-order interactions are relatively limited.

We acknowledge that this decomposition is a structural assumption rather than a universal property of nonlinear diffusion models. More generally, the conditional noise predictor can be expressed as ϵ∗(z,oti)=∑j∈Ctiwijϵ(z,otj)+rti\boldsymbol{\epsilon}^{\ast}(z,\mathbf{o}_t^i) = \sum_{j\in\mathcal{C}_t^i} w_{ij}\boldsymbol{\epsilon}(z,o_t^j) + r_t^i, where rtir_t^i represents the residual dependencies not captured by the weighted pairwise interaction decomposition. When higher-order interactions are sufficiently weak, this residual is expected to contribute only a limited approximation error. If the residual is bounded, it introduces an additional term into the noise prediction error bound, which can be incorporated into the subsequent trajectory-distribution error analysis under the corresponding regularity conditions. The original theoretical result is recovered when rti=0r_t^i=0.

Regarding the remaining assumptions, uniformly bounded history-representation errors, Lipschitz continuity, predictor smoothness, and bounded neighborhood dispersion are regularity conditions commonly adopted in theoretical approximation analyses. Following this established practice, we employ these assumptions to keep the analysis tractable and focused on the effects of history inference and MF approximation, rather than introducing an overly complex end-to-end characterization. We acknowledge that certain error sources, particularly finite-capacity neural network fitting errors, are not explicitly covered by the current theoretical bound. Additionally, as discussed in Appendix B.2, we adopt an early-stopping treatment to address the diffusion weighting singularity near the zero-noise endpoint. Overall, our theoretical analysis provides a principled characterization of how history representation and weighted MF approximation errors affect the reliability of local diffusion trajectory prediction under the stated assumptions.

References:

[1] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean Field Multi-Agent Reinforcement Learning,” in Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 5571–5580, 2018.

[2] Q. Hao, W. Huang, T. Feng, J. Yuan, and Y. Li, “GAT-MF: Graph Attention Mean Field for Very Large Scale Multi-Agent Reinforcement Learning,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 685–697, 2023.

Q2. Joint trajectory consistency.

We thank the reviewer for raising this important concern. Although MODAL employs decentralized local world models, their predictions are conditioned on neighboring agents' information through weighted MF communication. Specifically, distance-weighted aggregation incorporates neighborhood interactions, presence indicators, and density information into each agent's prediction context, enabling local models to capture inter-agent dependencies while maintaining fixed-dimensional communication. Furthermore, Theorem 1 establishes an upper bound on the discrepancy between predicted and true HH-step local trajectory distributions under the stated assumptions, explicitly accounting for the approximation error introduced by MF aggregation.

To quantitatively assess cross-agent consistency in imagined trajectories, we extend Table 1 with a Joint Trajectory Consistency (JTC) metric and provide additional results on MOSMAC 3t and 8t in Appendix E.2. JTC evaluates how accurately predicted trajectories preserve inter-agent spatial relationships by measuring discrepancies between predicted and ground-truth relative positions of neighboring agents. Specifically, let r^j∣it+h\hat{\mathbf{r}}_{j|i}^{t+h} and rj∣it+h\mathbf{r}_{j|i}^{t+h} denote the predicted and actual positions of agent jj relative to agent ii at future step t+ht+h, respectively. We define E_{\mathrm{JTC}} = \frac{1}{H} \sum_{h=1}^{H} \frac{1}{|\mathcal{P}_{t+h}|} \sum_{(i,j)\in\mathcal{P}_{t+h}} \left\\| \hat{\mathbf{r}}_{j|i}^{t+h} - \mathbf{r}_{j|i}^{t+h} \right\\|_2, where Pt+h\mathcal{P}_{t+h} denotes the set of valid neighboring-agent pairs, with an identical evaluation mask across methods. We evaluate JTC over 4- and 8-step rollouts using the same test segments and ground-truth action sequences without teacher forcing. Errors are averaged over evaluation segments and reported as mean ±\pm standard deviation across eight seeds. Lower JTC indicates better preservation of cross-agent spatial relationships.

Table R-1. World model prediction error and joint trajectory consistency on MOSMAC. Values are mean ±\pm standard deviation over eight seeds.

MethodLocal ℓ1\ell_1 Error (4-step)Local ℓ1\ell_1 Error (8-step)JTC Error (4-step)JTC Error (8-step)
MOSMAC 3t
MODAL0.406±0.019\mathbf{0.406}_{\pm\mathbf{0.019}}0.911±0.033\mathbf{0.911}_{\pm\mathbf{0.033}}0.1880±0.0117\mathbf{0.1880}_{\pm\mathbf{0.0117}}0.2173±0.0116\mathbf{0.2173}_{\pm\mathbf{0.0116}}
MODAL-G0.825±0.0330.825_{\pm0.033}1.553±0.0581.553_{\pm0.058}0.2941±0.02960.2941_{\pm0.0296}0.3050±0.03210.3050_{\pm0.0321}
MODAL-10.439±0.0280.439_{\pm0.028}1.014±0.0631.014_{\pm0.063}0.1957±0.01560.1957_{\pm0.0156}0.2272±0.01680.2272_{\pm0.0168}
DIMA0.579±0.0870.579_{\pm0.087}1.170±0.2321.170_{\pm0.232}0.3415±0.02170.3415_{\pm0.0217}0.3503±0.02330.3503_{\pm0.0233}
MOSMAC 4t
MODAL0.386±0.011\mathbf{0.386}_{\pm\mathbf{0.011}}0.889±0.021\mathbf{0.889}_{\pm\mathbf{0.021}}0.2001±0.0037\mathbf{0.2001}_{\pm\mathbf{0.0037}}0.2512±0.0039\mathbf{0.2512}_{\pm\mathbf{0.0039}}
MODAL-G0.884±0.0410.884_{\pm0.041}1.629±0.0531.629_{\pm0.053}0.3665±0.04970.3665_{\pm0.0497}0.4012±0.05460.4012_{\pm0.0546}
MODAL-10.429±0.0110.429_{\pm0.011}1.016±0.0241.016_{\pm0.024}0.2101±0.00680.2101_{\pm0.0068}0.2672±0.01300.2672_{\pm0.0130}
DIMA0.599±0.0590.599_{\pm0.059}1.180±0.1411.180_{\pm0.141}0.3486±0.00830.3486_{\pm0.0083}0.3693±0.01030.3693_{\pm0.0103}
MOSMAC 8t
MODAL0.412±0.014\mathbf{0.412}_{\pm\mathbf{0.014}}0.877±0.040\mathbf{0.877}_{\pm\mathbf{0.040}}0.2834±0.0374\mathbf{0.2834}_{\pm\mathbf{0.0374}}0.3343±0.0339\mathbf{0.3343}_{\pm\mathbf{0.0339}}
MODAL-G0.697±0.0290.697_{\pm0.029}1.302±0.0431.302_{\pm0.043}0.4776±0.02040.4776_{\pm0.0204}0.4939±0.01690.4939_{\pm0.0169}
MODAL-10.435±0.0510.435_{\pm0.051}0.982±0.1620.982_{\pm0.162}0.3106±0.05160.3106_{\pm0.0516}0.3625±0.03850.3625_{\pm0.0385}
DIMA0.585±0.0520.585_{\pm0.052}1.125±0.1021.125_{\pm0.102}0.4322±0.00720.4322_{\pm0.0072}0.4424±0.00690.4424_{\pm0.0069}

As shown in Table R-1, MODAL consistently achieves the lowest local prediction and JTC errors across all three scenarios and both rollout horizons. Compared with DIMA, MODAL reduces JTC errors by 24.4%–44.9% across the six evaluation settings. Notably, its advantage remains pronounced in the more challenging 8t scenario, demonstrating that decentralized local world models can effectively preserve inter-agent spatial relationships as coordination complexity increases. These results provide complementary empirical evidence to Theorem 1, which bounds local trajectory distribution errors, and the weighted MF ablation in Fig. 6. Together, they support the effectiveness of MODAL in capturing inter-agent dependencies and generating spatially coherent imagined trajectories for cooperative policy learning.

Q3. Direct validation of temporal objective aliasing.

We thank the reviewer for this suggestion. In the newly added Appendix E.3 Controlled Temporal Objective Aliasing Evaluation, we construct 26 history pairs from independent MOSMAC 12t episodes, with similar current observations (RMS ≤0.04\leq 0.04) but distinct histories (RMS ≥0.05\geq 0.05). For each pair, we evaluate five actions valid in both contexts, with eight repetitions per action. Each action is followed by the same fixed policy for seven steps, yielding 2,080 real-environment rollouts. We compare RMS distances between 8-step continuation-value profiles with Euclidean distances between learned history representations, using identical history pairs and reference values across methods, without training additional predictors. Spearman's rank correlation (ρ∈[−1,1]\rho\in[-1,1]) measures their monotonic association: positive, negative, and zero values indicate positive, negative, and no monotonic association, respectively. Higher positive values indicate better alignment.

As shown in Table R-2, MODAL achieves the highest correlation, indicating better preservation of continuation-value differences. Furthermore, Fig. 11 in the revised manuscript's Appendix E.3 Controlled Temporal Objective Aliasing Evaluation compares policy performance: IPPO is model-free; History-PPO+MF retains the same historical and neighborhood inputs without model-based imagination; and capacity-matched Gaussian WM replaces diffusion with a Gaussian predictor of comparable capacity. MODAL learns faster on MOSMAC 8t and achieves stronger performance on the challenging 12t, supporting the benefits of history-conditioned diffusion modeling beyond recurrent memory and Gaussian world models.

Table R-2. Spearman correlation between continuation-value and history-representation distances on MOSMAC 12t.

MethodSpearman ρ\rho ↑\uparrow
MODAL0.410
History-PPO+MF0.198
Capacity-matched Gaussian WM0.037

Q4. Computational efficiency.

We thank the reviewer for raising this issue. We compare training time over 10710^7 environment steps, deployment parameters, inference latency, and peak inference GPU memory on MOSMAC 3t and 4t, adding MARIE [3] and DMAWM [4] to the original baselines. All experiments use a single NVIDIA RTX 4090 under consistent settings.

Table R-3 shows that MODAL trains more slowly than lightweight model-free methods and some model-based alternatives, but much faster than MAMBA and MARIE. Its policy-only execution avoids diffusion sampling, requiring far fewer deployment parameters than MAMBA, MARIE, and DMAWM (comparable to DIMA), lower latency than those three methods, and less inference memory than MAMBA and DMAWM. Although some baselines are cheaper, MODAL offers a favorable training–deployment trade-off. These results are incorporated into the newly added Section 6.6 Computational Efficiency of the revised manuscript.

References:

[3] Y. Zhang, C. Bai, B. Zhao, J. Yan, X. Li, and X. Li, “Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models,” Transactions on Machine Learning Research (TMLR), 2025.

[4] D. Xue, J. Jiang, S. Zhang, W. Guo, L. Yuan, Z. Zhang, and Y. Yu, “Learning Disentangled Multi-Agent World Model for Decentralized Control,” in Proceedings of the 43rd International Conference on Machine Learning (ICML), pp. 143194–143215, 2026.

Table R-3. Computational efficiency on MOSMAC 3t and 4t.

MethodTrain Time (h)Deploy Params (M)Latency (ms)Peak Inf. Mem. (MiB)
MOSMAC 3t
MODAL8.280.0381.03116.39
IPPO3.640.0280.3089.25
MAPPO5.070.0070.3079.17
IA2C2.980.0280.3489.25
MAMBA83.4511.6262.56468.42
MARIE34.221.3411.26714.80
DMAWM7.5137.3452.346157.22
DIMA5.060.0390.6999.28
MOSMAC 4t
MODAL8.360.0400.89216.44
IPPO3.820.0290.3539.25
MAPPO5.350.0080.3399.18
IA2C2.830.0290.3299.25
MAMBA70.5711.6352.47568.50
MARIE34.441.3671.21514.92
DMAWM6.9937.3681.847157.36
DIMA4.420.0400.7279.28

Comments on the paper

1 comment
Area_Chair_hHThOpenReview ↗

Please start the discussion on the rebuttals

Dear Reviewers,

Thanks a lot for your help reviewing this submission! We have received the rebuttals of the authors, so I would like to hear everyone's opinion after the authors' rebuttal to help me form my final meta-review and recommendation for this paper.

Can I ask everyone to share a little bit about their post-rebuttal feelings about the paper by replying to this message thread, and update their original review/rating of the paper accordingly at your earliest convenience? Thanks a lot in advance!

Best, your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.