acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models

Abstract

Model-based offline multi-agent reinforcement learning (MARL) provides a promising approach to multi-agent decision-making under partial observability, where the global state is unavailable from offline data. However, limited dataset coverage can lead to substantial out-of-distribution (OOD) model errors, which may mislead policy optimization and degrade downstream performance. Existing model-based offline MARL methods typically rely on access to global states or mitigate model errors through pessimistic or conservative policy optimization, leaving the reliability and refinement of learned world models less explored. We propose a Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data. Building on posterior information loss, we extend this criterion to the multi-agent setting as multi-agent posterior information loss (MPIL), which provides a reliability measure for world model refinement and offline evaluation. Theoretically, we establish an upper bound on policy regret that is monotonic in MPIL and further characterize its dependence on the size and state–joint-action coverage of the offline dataset. Experiments on partially observable multi-agent variants of D4RL MuJoCo tasks demonstrate competitive policy performance across diverse offline data conditions, while MPIL-guided refinement improves world model prediction and downstream policy performance. These results support a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.

Primary area:
reinforcement learning
Keywords:
Offline Multi-Agent Reinforcement Learning, Regret Estimation, Posterior Information Loss, Latent World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4KYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a Bayesian model-based offline MARL framework under partial observability. It uses a GRU to infer latent states from observation-action histories and learns a world model ensemble. The authors introduce MPIL to guide world model refinement and establish regret bounds in terms of MPIL. Experiments on multi-agent MuJoCo tasks show competitive performance.

Critical strengths
  1. I think the paper addresses an important problem in offline MARL, where the global state is unavailable and model errors can affect policy learning.

  2. The proposed method achieves competitive performance across different datasets, and the experiments show that world model refinement can improve prediction accuracy and policy performance.

Critical weaknesses
  1. I have concern about the novelty of the paper. Most algorithmic components are standard. In particular, Proposition 4.1 shows that MPIL differs from NLL only by a constant, so the proposed refinement is essentially incremental NLL training with early stopping. The regret analysis also appears closely related to SOReL. The paper does not clearly establish its contributions beyond existing methods.

  2. In theory, eq.(1) defines the next latent state as a deterministic function of the next observation, while Assumption A.1 assumes a factorized transition-observation model. It is unclear whether these two are compatible. Moreover, the predictive sufficiency of the learned latent states is assumed rather than guaranteed by the proposed GRU encoder. These issues raise concerns about whether the theoretical guarantees apply to the actual algorithm.

  3. The experiments lack important ablation studies: since the joint observations recover the original D4RL observation, it is unclear whether GRU-based latent-state inference is necessary. Similarly, the paper does not compare MPIL-based refinement against standard NLL training under the same computational budget. Without these comparisons, it is difficult to determine which components are responsible for the reported improvements.

Questions for authors
  1. What is the main advantage of MPIL-based refinement over standard NLL training with early stopping? Can the authors provide a comparison under the same computational budget?

  2. How is the factorization in Assumption A.1 compatible with the GRU state update in eq. (1)? Does Theorem 5.1 apply to the implemented world model?

  3. Since the joint observations recover the original D4RL observation, I'm curious why is GRU-based latent-state inference necessary. How does the method perform when directly using the joint observations?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for raising this concern. We agree that the individual optimization components are largely standard. Under the conditions of Proposition 4.1, MPIL differs from NLL by a model-independent entropy term, so the two objectives have the same gradients and minimizers. We do not present MPIL as a new optimization algorithm. Its contribution is to give standard NLL training a policy-oriented interpretation: under the stated conditions, changes in NLL correspond to changes in posterior-averaged model discrepancy, which Theorem 5.1 relates to joint-policy regret under its own assumptions.

Our conditional analysis extends the perspective of SOReL's single-agent observed-state setting (Fellows et al., 2026) to latent-state cooperative multi-agent learning. MPIL accounts for errors in latent-state transitions, shared rewards, and local-observation generation, while the performance analysis considers joint actions and decentralized policies. SOReL measures model error in the state–action regions visited by a policy. Our MPIL instead measures model error on the latent-state–joint-action pairs in the offline data used to train the world model. To connect error on the offline data to error in the regions visited by the joint policy, we introduce a coverage condition and obtain Iπ~≤CID^N\mathcal{I}^{\tilde{\pi}} \leq C\mathcal{I}_{\hat{D}_N}. The resulting conditional regret bound therefore depends on both offline-data model error and coverage of the regions visited by the joint policy. We have clarified the relationship to SOReL and the scope of this extension to latent-state multi-agent learning in the revised manuscript.

The experiments also test the practical value of staged refinement. Section 6.3 and Fig. 3 compare initial, refined, and Mixed world models on Walker2d. The Mixed WM is a conventional one-shot NLL reference at the same total sample budget of 0.5M transitions. Across the four Walker2d dataset types, the refined models lead to higher downstream policy returns than the corresponding initial models. These results support the practical benefit of refinement, while the shared likelihood objective remains unchanged. The Mixed WM uses a separately drawn set of 0.5M mixed samples, so the Mixed and refined models have the same total sample count but different training datasets. We therefore do not attribute this comparison exclusively to the order in which samples are introduced. The current experiments do not establish a matched total gradient-update budget. Our complete method also achieves higher mean returns than MOMA-PPO (Barde et al., 2024) in 10 of the 12 settings in Table 1; that comparison concerns complete methods rather than refinement alone.

References

• Paul Barde, Jakob N. Foerster, Derek Nowrouzezahrai, and Amy Zhang (2024). A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141–150.

• Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel, Michael A. Osborne, and Jakob Nicolaus Foerster (2026). Fully Offline Reinforcement Learning. Reinforcement Learning Journal, volume 7.

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2 Response

Eq. (1) describes the recursive GRU encoder used in the implementation, whereas Assumption A.1 specifies the idealized latent-state structure used in the theoretical analysis. Because the next latent state is computed from the next observation, the factorization in Assumption A.1 is not implied by the GRU update and requires an additional structural condition when the next observation remains conditionally stochastic. In the revised manuscript, we have made this distinction explicit and presented Theorem 5.1 as a conditional analysis under predictive sufficiency, kernel factorization, finite KL divergence, and coverage.

Predictive sufficiency is imposed as a representation assumption in our theoretical analysis: the latent state is assumed to preserve the information from the full history that is relevant to future prediction. The GRU reconstruction objective encourages the latent state to retain observation information but is not used to establish this assumption. Under the stated conditions, Theorem 5.1 provides a multi-agent performance analysis that incorporates errors in latent-state transitions, shared rewards, and local-observation generation into MPIL and connects these errors, together with offline-data coverage, to joint-policy regret. The implemented GRU world model is evaluated empirically through its predictive accuracy and downstream policy performance. Extending the analysis to approximately sufficient representations, for example through the approximate information-state framework, remains an important direction for future work.

For the implemented GRU–Gaussian model, these conditions are not satisfied: dependence of the next latent state on the next observation conflicts with the factorization when it is non-degenerate, and a degenerate latent transition is singular with respect to the Gaussian model. Thus Theorem 5.1 does not certify the implemented MuJoCo policies. Appendix A.3 now makes this limitation explicit.

AuthorsOpenReview ↗

Response to Weakness 3

Thank you for this suggestion. We have added a latent-state ablation and expanded the world-model refinement comparisons. We have also corrected our inaccurate statement that concatenating local observations recovers the original D4RL observation. In our implementation, the concatenated local observations differ from the original D4RL observation in both dimensionality and composition.

(1) We replaced the GRU latent state with the concatenation of the agents' current local observations while keeping the remaining world-model architecture, refinement procedure, decentralized actor inputs, and policy-learning settings unchanged. As shown in Table 2, the latent-state world model achieves lower mean observation-prediction RMSE in all 12 task–dataset settings. For example, the RMSE decreases from 0.1190 to 0.0270 on Hopper-random and from 0.4277 to 0.3311 on Walker2d-full-replay. The latent-state model also leads to higher mean downstream policy returns in every setting. Representative improvements include an increase from 22.39 to 43.32 on HalfCheetah-full-replay and from 51.65 to 63.57 on Walker2d-full-replay. These results show that integrating observation–action history through the GRU provides a more predictive representation than directly concatenating the current local observations, and that this benefit transfers to downstream policy learning.

(2) Proposition 4.1 establishes that MPIL and NLL are optimization-equivalent under its stated conditions; therefore, both variants are implemented through likelihood training. Section 6.3 and Fig. 3 compare the initial, refined, and Mixed world models and their downstream policies. The Mixed WM is the conventional one-shot NLL reference, trained on 0.5M samples from the mixed pool. Each refined model is initialized on 0.4M samples of one dataset type and refined with 0.1M additional samples from the mixed pool. Across the four Walker2d dataset types, the refined world models lead to higher downstream returns than the corresponding initial models. These results provide evidence of practical benefits from refinement and include a one-shot NLL reference with the same total number of samples. The Mixed WM uses a separately drawn set of 0.5M mixed samples, rather than the same final samples used by each refined model. This is a comparison at the same total sample count with different training datasets, so it does not isolate the effect of staged sample introduction. The current experiments do not establish a matched total gradient-update budget.

The latent-state ablation isolates the contribution of history-based inference. The refinement results show the performance of the staged procedure relative to the initial and one-shot Mixed references, subject to the data-composition and computation distinctions described above.

Reviewer_z7sfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a Bayesian world-model framework for offline multi-agent reinforcement learning under partial observability. It infers latent global states from local observation-action histories and uses a model ensemble to represent uncertainty in the learned environment dynamics. The main idea is to extend posterior information loss to the multi-agent setting (MPIL) and use it to guide world-model refinement before training policies through imagined interactions. The paper also provides a regret analysis connecting world-model reliability, offline data size and coverage, and policy performance. Experiments on multi-agent variants of D4RL MuJoCo tasks demonstrate competitive returns against the evaluated baselines, while the refinement experiments show improvements in both model prediction accuracy and downstream policy performance.

Critical strengths
  1. The method follows a clear and relatively simple idea: improving the reliability of the learned world model before using it for policy optimization. Extending PIL to the multi-agent setting and using MPIL to guide model refinement gives the framework a coherent motivation. The regret analysis further connects model discrepancy to policy performance under the stated assumptions, providing theoretical support for this design.

  2. The experimental results demonstrate promising practical benefits. The method achieves the highest reported return in eight of the twelve settings in Table 1, with particularly clear improvements on Walker2d-full-replay and Hopper-random. Figure 3 further shows that world-model refinement improves both prediction accuracy and downstream policy performance, supporting the usefulness of improving the model before policy learning.

Critical weaknesses
  1. The experimental discussion provides limited insight into why the gains vary across tasks and dataset qualities. Further analysis would help clarify the method's strengths and limitations.

  2. The current comparisons leave some uncertainty about the method's competitiveness against more recent offline MARL approaches. Including a representative baseline such as ComaDICE (ICLR 2025), under matched observation and dataset settings, would help clarify this.

Questions for authors
  1. Could the authors provide more insight into the task-dependent results? For example, why does MOMA-PPO perform better on HalfCheetah-medium and full-replay, and what might explain the substantial performance deterioration of OMAR in some of the expert-data learning curves?

  2. Could the authors include an additional representative recent offline MARL baseline, such as ComaDICE (ICLR 2025), under the same observation and dataset settings? A comparison on a few representative tasks would help clarify the method's competitiveness. If there are practical obstacles to a fair comparison, please explain them.

4 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for the suggestion. We have expanded Section 6.1 and the appendix to discuss task- and dataset-dependent performance using Table 1, the world-model refinement experiments, and related offline MARL studies.

(1) Advantages on non-expert data. Our method scores 31.16 and 24.38 on Hopper-random and HalfCheetah-random, respectively, versus 3.24 and 2.25 for BC. These results suggest that it can exploit dynamics and reward information even from low-return trajectories. On Walker2d-full-replay, our method achieves 63.57 versus 26.52 for MOMA-PPO, while the refinement experiment shows that lower prediction error is accompanied by higher policy returns.

(2) Task- and dataset-dependent limitations. All methods obtain low returns on Walker2d-random. Although our method performs best, the limited improvement is consistent with insufficient policy-relevant coverage. BC performs better on Hopper-expert and Walker2d-expert because high-quality demonstrations directly support imitation. However, this relationship is not monotonic under partial observability: on HalfCheetah, BC scores 44.62 on medium data but only 5.59 on expert data. Compounding errors (Ross and Bagnell, 2010) and limited recovery-behavior coverage can make narrowly distributed expert data less robust. A similar pattern has been reported for partially observable ITD3+BC (Barde et al., 2024). On HalfCheetah-medium and full-replay, MOMA-PPO achieves higher returns than our method. These results illustrate that lower average world-model prediction error does not necessarily translate into higher policy returns. Final performance also depends on model accuracy in the regions actually visited by the policy and on whether the policy optimizer can effectively exploit model-generated data. Moreover, MOMA-PPO uses MAPPO, whereas our method uses SAC with conservative regularization; the difference between these policy-optimization mechanisms also affects the final returns.

(3) Low OMAR performance. Similarly low OMAR returns have been reported under different multi-agent MuJoCo configurations (Wang et al., 2023). OMAR relies on local conservative Q-functions to evaluate candidate actions and rectify policies. Under partial observability, these local estimates may not fully capture joint coordination, while conservative regularization can prevent the policy from escaping suboptimal offline behaviors.

Overall, the results suggest that the method’s benefits depend on coverage of policy-relevant behaviors; gains can be limited when coverage is poor or when high-quality demonstrations already support effective imitation.

References

• Paul Barde, Jakob N. Foerster, Derek Nowrouzezahrai, and Amy Zhang (2024). A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141–150.

• Stéphane Ross and J. Andrew Bagnell (2010). Efficient Reductions for Imitation Learning. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9, pp. 661–668.

• Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan (2023). Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization. Advances in Neural Information Processing Systems, 36:52413–52429.

AuthorsOpenReview ↗

Response to Weakness 2

We have added ComaDICE (Bui et al., 2025) as a recent representative baseline and reported the results in Table 1. All methods use the same multi-agent task partitions, local-observation configurations, offline datasets, evaluation protocol, and return normalization, without access to the simulator's global state. Our method achieves higher average returns than ComaDICE in 11 of the 12 task–dataset settings. For example, it achieves 31.16 versus 20.34 on Hopper-random, 24.38 versus 18.03 on HalfCheetah-random, and 63.57 versus 58.29 on Walker2d-full-replay. After including ComaDICE, our method still achieves the highest average return in 8 of the 12 settings, supporting its competitiveness across tasks and dataset qualities. Appendix A.2.4 describes the ComaDICE procedure and reports its hyperparameters; Appendix A.2.5 specifies the common local-observation configuration.

References

• The Viet Bui, Thanh Hong Nguyen, and Tien Mai (2025). ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization. International Conference on Learning Representations.

AuthorsOpenReview ↗

Response to Question 1

Thank you for raising these task-specific questions. Section 6.1 and the baseline-performance discussion in the appendix address these patterns. We discuss plausible explanations below.

(1) HalfCheetah-medium and full-replay. MOMA-PPO (Barde et al., 2024) achieves 50.35 and 50.13, compared with 46.09 and 43.32 for our method. As discussed in our response to Weakness 1, final performance depends on both world-model quality and policy optimization. MOMA-PPO performs MAPPO updates on model-generated trajectories, controlling update risk through PPO clipping, model-uncertainty penalties, and rollout truncation. Our method instead uses off-policy SAC (Haarnoja et al., 2018) with conservative Q-regularization, which lowers the estimated values of actions weakly supported by the offline data. These differences may allow MOMA-PPO to make fuller use of model-generated data, whereas our conservative regularization may restrict the range of policy updates. Moreover, lower average prediction error does not necessarily imply greater accuracy in the regions actually visited by the learned policy. These differences may help explain the higher returns of MOMA-PPO in these two settings. Nevertheless, our method outperforms MOMA-PPO in 10 of the 12 settings, and the Walker2d refinement experiments show higher policy returns after refinement across all four dataset types. These results support the overall value of world-model refinement without implying superiority in every setting.

(2) OMAR degradation on expert data. OMAR (Pan et al., 2022) uses learned Q-values to rank candidate actions and rectify policies. Expert datasets provide high-quality demonstrated actions but do not guarantee accurate value estimates for candidate actions that deviate from those demonstrations. This issue is amplified in multi-agent settings: an action-ranking error by one agent changes the joint action and can disrupt the coordinated behavior of the other agents, while errors across agents may compound and move the joint policy outside the coordinated regions supported by the data. The policy may therefore drift toward actions with high estimated values but poor joint execution, gradually disrupting previously effective behavior. This possible mechanism is consistent with the return deterioration observed on Hopper-expert and Walker2d-expert.

References

• Paul Barde, Jakob N. Foerster, Derek Nowrouzezahrai, and Amy Zhang (2024). A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141–150.

• Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the International Conference on Machine Learning, pp. 1861–1870.

• Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu (2022). Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification. Proceedings of the International Conference on Machine Learning, pp. 17221–17237.

AuthorsOpenReview ↗

Response to Question 2

As discussed in our response to Weakness 2, we have added ComaDICE (Bui et al., 2025) as a recent representative baseline. The comparison covers three tasks and four dataset types, resulting in 12 settings. All methods use the same observation configurations, offline datasets, and evaluation protocol, as reported in Table 1. Our method achieves higher average returns than ComaDICE in 11 of the 12 settings, further supporting its competitiveness against recent offline MARL methods.

References

• The Viet Bui, Thanh Hong Nguyen, and Tien Mai (2025). ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization. International Conference on Learning Representations.

Reviewer_wQTVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an MPIL-guided Bayesian world model method for offline cooperative multi-agent RL under partial observability, where the global state is not available in the data. It infers a latent global state from joint observation-action histories, and learns an ensemble of transition, reward and observation models as an approximate posterior. It extends posterior information loss to the multi-agent setting (MPIL), and shows that MPIL equals the posterior-averaged negative log-likelihood up to a model-independent constant. Theoretically, the regret of the policy that is Bayes-optimal under the learned world model is bounded by an increasing function of MPIL. If MPIL is assumed to shrink at the parametric rate, this gives an expected regret rate of O(1/sqrt(N)), where N is the dataset size. Empirically, on multi-agent variants of D4RL MuJoCo tasks, the method is competitive with model-free and model-based baselines.

Critical strengths

The paper extends posterior information loss to partially observable MARL and relates its changes to changes in validation NLL, enabling offline monitoring of model refinement. In the experiments, the method achieves the highest mean return in eight of twelve task-dataset settings and remains competitive in the others.

Critical weaknesses
  1. The stopping criterion does not show that the model is reliable enough for policy learning, which is a central claim of the paper. Refinement stops when the change in MPIL between stages is small (Eq. 9), but Theorem 5.1 depends on the MPIL value and data coverage. A model can stop improving while its prediction errors are still large. So a small change implies neither a small regret bound nor sufficient offline data. The value of MPIL itself also cannot be recovered from the monitored NLL alone, since Proposition 4.1 relates them through an unknown entropy term. Moreover, this identity assumes a fixed data distribution. If the encoder is updated during refinement, keeping the raw validation trajectories fixed does not necessarily keep the latent validation distribution or its entropy fixed. The paper should clarify whether the encoder is frozen and how this condition is maintained. Thus, satisfying the refinement criterion alone does not establish that the model is reliable in the true environment.

  2. The experiments do not show where the gains in Table 1 come from. The method is motivated by the absence of the global state, but the union of the local observations reconstructs the full D4RL observation. So the experiments do not establish the benefit of recurrent latent-state inference over directly using the combined observations. Also, the policy learner mixes imagined and real data, uses rollouts of length one, and adds a conservative loss. This makes it important to separate the benefit of world model refinement from that of conservative policy optimization. There is no ablation that removes the conservative loss or replaces the recurrent latent with the concatenated observations. The baseline results also need further explanation. Baseline hyperparameters are reported only for Walker2d-random (Table 6). OMAR scores below 2 in 5 out of the 8 Hopper and Walker2d settings, and BC performs better in HalfCheetah-medium (44.62) than HalfCheetah-expert (5.59). The paper should explain these performance patterns and provide further details of baseline implementation and tuning. Without these ablations and clarification of the baseline implementations and tuning, the results do not isolate the contribution of MPIL-guided world model refinement.

Questions for authors
  1. In Section 6.3, are the added subsets drawn from the same dataset type or from a mixed pool? How many samples and gradient steps does the refined model use compared with the baseline trained on 0.5M samples, and is the encoder fine-tuned during refinement? Please compare with a model trained once on the final data with the same compute.

  2. Is MPIL (Eq. 5) finite for the models in the experiments? The MuJoCo dynamics are deterministic, and the 64-dimensional latent state is a deterministic function of a lower dimensional observation history (Table 2). Am I correct that the true one-step kernel is then a point mass, so its KL divergence from the Gaussian ensemble members would be infinite and the bound in Theorem 5.1 would be trivial? If so, what does Theorem 5.1 guarantee for the implemented method?

  3. How were OMAR and MOMA-PPO tuned for each task? Why does OMAR score below 2 in most Hopper and Walker2d settings, and why does BC score 5.59 on HalfCheetah-expert but 44.62 on HalfCheetah-medium (Table 1)? Also, Section 4.3 states that each actor uses its local observation history, but Appendix A.2.5 states that actor i receives only its current local observation. Which input do the actors use, and what do the critics and world models of the baselines receive? Section 6 states that the global state is unavailable to all methods, but the joint observation is the full D4RL observation. Then, do the results hold against a model-based single-agent style baseline such as MOPO or COMBO with the same ensemble and decentralized actors?

  4. In Fig. 2, which data make up the dataset sizes up to 4M? Are these unique transitions aggreagted from different dataset types, or what is the composition at each dataset size?

  5. Section 4.1 does not describe how the latent state in Eq. 1 is initialized at the start of an episode. Table 2 lists a sequence length of 32, but the paper does not say whether the hidden state is reset at the start of each 32-step window or carried over from the previous window. How are the latent states in the state-augmented dataset (Section 4.1) computed for steps late in an episode, for example step 500? Is the encoder run from the first step of the episode, or only over the last 32 steps? Table 2 also lists a fine-tuning learning rate. If the encoder is fine-tuned after the data are mapped into the latent space, are these latent states recomputed? Finally, do imagined rollouts start from latent states stored in this dataset? If so, with a rollout length of 1, the world model predicts only one step from a latent state computed on real data, and the recurrent encoder is not used during imagination?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We agree. Satisfying the stopping criterion does not establish that the world model is reliable in the true environment, and a small change between stages implies neither a small MPIL nor a small regret bound. We have revised the abstract, introduction, Section 4, and Section 6.2 to distinguish the empirical stopping statistic from a guarantee of model accuracy or sufficient data coverage.

(1) The stopping rule only detects small changes in held-out validation NLL as more offline data are added. It is a saturation signal that indicates diminishing changes in the objective on the available data. It does not show that the model is accurate or that the data are sufficient for policy learning. A plateau may also reflect limited model capacity or optimization difficulties, which the rule cannot distinguish from data saturation. Section 4.3 now states this limitation explicitly.

(2) We agree that the absolute value of MPIL cannot be recovered from the monitored NLL, for the same reason that a KL divergence cannot be recovered from a cross-entropy without the entropy of the true distribution. The implemented criterion monitors changes in NLL rather than absolute MPIL. Proposition 4.1 now states the common-reference-measure and integrability conditions under which population NLL differences equal MPIL differences, with the representation and evaluation distribution fixed. These conditions do not hold for the implemented GRU–Gaussian model, as discussed in our response to Question 2. In our experiments, the refinement statistic is therefore the change in held-out NLL itself.

Whether stopping at an NLL plateau leads to good policies is an empirical question, examined in Figs. 2 and 3, rather than a consequence guaranteed by the stopping rule.

(3) The encoder is frozen during refinement. The fine-tuning learning rate in the submitted hyperparameter table was an error: the latent-state inference model is trained once at a learning rate of 3 × 10⁻⁴, and no fine-tuning stage exists. After training, the encoder is frozen, and all data subsets, including those added during refinement, are mapped into the latent space with this frozen encoder. Refinement updates only the ensemble parameters Φ. With the encoder frozen and the validation data fixed, the held-out NLL is evaluated in a consistent latent representation across refinement stages. Its change serves as the practical refinement statistic. These details are specified in Section 4.1 and Appendix A.2.1.

AuthorsOpenReview ↗

Response to Weakness 2

Thank you for raising these concerns. We have corrected the observation description and added ablation results to clarify the contributions of latent-state inference and conservative policy optimization.

(1) We replaced the GRU latent state with the concatenation of the agents' current local observations while keeping the remaining world-model architecture, refinement procedure, decentralized actor inputs, and policy-learning settings unchanged. As shown in Table 2, the latent-state world model achieves lower mean observation-prediction RMSE in all 12 task–dataset settings. For example, the RMSE decreases from 0.1190 to 0.0270 on Hopper-random and from 0.4277 to 0.3311 on Walker2d-full-replay. The latent-state model also leads to higher mean downstream policy returns in every setting. Representative improvements include an increase from 22.39 to 43.32 on HalfCheetah-full-replay and from 51.65 to 63.57 on Walker2d-full-replay. These results show that integrating observation–action history through the GRU provides a more predictive representation than directly concatenating the current local observations, and that this benefit transfers to downstream policy learning.

(2) To isolate the contribution of conservative policy optimization after world-model refinement, we set the conservative-loss weight to zero while keeping the refined world model fixed and all other policy-learning settings unchanged. As reported in Section 6.1, the mean normalized return over five random seeds decreases from 63.57 ± 11.93 to 50.44 ± 6.81 on Walker2d-full-replay and from 46.09 ± 3.40 to 27.32 ± 1.36 on HalfCheetah-medium. These results show that conservative regularization improves mean policy performance in these two settings, even when the refined world model is held fixed.

(3) Section 6.3 and Fig. 3 report world-model prediction errors and downstream policy returns for the initial, refined, and Mixed world models. The Mixed WM provides a conventional one-shot NLL reference. The refinement comparisons retain the same policy-learning setup, including conservative regularization. The conservative-loss ablation above provides a separate comparison with the refined model fixed; neither comparison alone establishes that all gains come from world-model refinement.

(4) We have included the hyperparameter settings for all baselines across all 12 task–dataset combinations in Appendix A.2.4 to improve reproducibility and transparency.

(5) Low OMAR returns have also been reported in decentralized multi-agent MuJoCo under different agent configurations (Wang et al., 2023). This provides contextual evidence rather than a directly matched comparison. OMAR relies on learned Q-values to rank candidate actions and rectify policies; under partial observability, inaccurate rankings can disrupt joint coordination. We discuss this interpretation in the baseline-performance paragraph of the appendix.

(6) Under decentralized partial observability, higher-quality demonstrations do not necessarily yield better BC performance. Compounding errors (Ross and Bagnell, 2010) can be exacerbated by limited state coverage and missing recovery behaviors. A related non-monotonic pattern has been reported for ITD3+BC under partial observability (Barde et al., 2024). This provides context for the observed BC scores of 44.62 on HalfCheetah-medium and 5.59 on HalfCheetah-expert, rather than establishing a unique cause of the difference. The revised appendix includes this discussion.

(7) The rollout length of 1 in the submitted table was a typo. All reported experiments use a rollout length of 5, as clarified in our response to Question 5 and Table 6.

References

• Paul Barde, Jakob N. Foerster, Derek Nowrouzezahrai, and Amy Zhang (2024). A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141–150.

• Stéphane Ross and J. Andrew Bagnell (2010). Efficient Reductions for Imitation Learning. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9, pp. 661–668.

• Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan (2023). Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization. Advances in Neural Information Processing Systems, 36:52413–52429.

AuthorsOpenReview ↗

Response to Question 1 and Question 2

Question 1 Response

The additional samples in Section 6.3 come from a mixed pool formed by concatenating the Walker2d random, medium, full-replay, and expert datasets, totaling 3,946,063 transitions. The encoder remains frozen throughout refinement.

For each dataset type, the world model is initialized on a subset of that dataset and refined using additional samples drawn from the mixed pool. Taking random as an example, the base model is trained on 0.4M random samples with 123,400 gradient updates and then refined with 0.1M additional samples from the mixed pool, reaching 0.5M samples after 78,150 additional gradient updates (201,550 updates in total). For reference, the Initial WM in Fig. 3 is trained once on 0.5M random samples, and the Mixed WM is trained once on 0.5M samples from the mixed pool.

The Mixed WM is the conventional one-shot NLL comparison in Fig. 3. The figure also reports policies trained with the initial, refined, and Mixed world models under the same policy-learning setup. Across the four Walker2d dataset types, the refined models lead to higher downstream returns than the corresponding initial models. The Mixed WM uses a separately drawn set of 0.5M mixed samples, rather than the same final samples used by the refined model. Thus, the two models use the same number of training samples but different sample compositions. This comparison does not isolate the effect of training order. The current experiments do not establish a matched total gradient-update budget.

Question 2 Response

Thank you for this careful question. We agree that the stated guarantee does not directly apply to the implemented model.

In our MuJoCo experiments, the dynamics and rewards are deterministic, and the next latent state is produced by the recurrent encoder as a deterministic function of the current latent state, joint action, and next joint observation. As Appendix A.3 now explains, if the next latent state is conditionally non-degenerate, its dependence on the next observation prevents the factorization assumed in Assumption A.1; if it is degenerate, its point-mass distribution is singular with respect to a non-degenerate Gaussian model. Moreover, under the implemented smooth GRU update, the next latent state is supported on the image of the lower-dimensional next-observation space, which has zero Lebesgue measure in the 64-dimensional latent space. Thus the finite-KL conditions needed for the guarantee are not satisfied by the implemented GRU–Gaussian model, and Theorem 5.1 gives no nontrivial guarantee for these models. A positive model variance does not change this support mismatch.

Theorems 5.1 and 5.2 are conditional results under the assumptions stated in Section 5 and Appendix A.3. Predictive sufficiency, kernel factorization, finite KL divergence, and coverage are additional assumptions; they are not implied by the recurrent encoder. The theorems do not certify the MuJoCo policies, and we do not claim that they do.

It minimizes and monitors the empirical posterior-averaged NLL, which is finite and well defined for the observed samples under the Gaussian models. Proposition 4.1 equates differences in the population likelihood objective with differences in MPIL only under its stated common-reference-measure and integrability conditions. In the experiments, the refinement statistic should be read as the change in held-out NLL.

The evidence supporting this criterion on MuJoCo is therefore empirical. Fig. 2 shows broadly consistent trends between the refinement statistic and the regret measures, while Fig. 3 reports prediction and policy improvements after refinement. The theory motivates the criterion but does not guarantee it on these tasks. Extending the analysis to deterministic dynamics with a suitable discrepancy measure remains a direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.