- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a Bayesian model-based offline MARL framework under partial observability. It uses a GRU to infer latent states from observation-action histories and learns a world model ensemble. The authors introduce MPIL to guide world model refinement and establish regret bounds in terms of MPIL. Experiments on multi-agent MuJoCo tasks show competitive performance.
Critical strengths
-
I think the paper addresses an important problem in offline MARL, where the global state is unavailable and model errors can affect policy learning.
-
The proposed method achieves competitive performance across different datasets, and the experiments show that world model refinement can improve prediction accuracy and policy performance.
Critical weaknesses
-
I have concern about the novelty of the paper. Most algorithmic components are standard. In particular, Proposition 4.1 shows that MPIL differs from NLL only by a constant, so the proposed refinement is essentially incremental NLL training with early stopping. The regret analysis also appears closely related to SOReL. The paper does not clearly establish its contributions beyond existing methods.
-
In theory, eq.(1) defines the next latent state as a deterministic function of the next observation, while Assumption A.1 assumes a factorized transition-observation model. It is unclear whether these two are compatible. Moreover, the predictive sufficiency of the learned latent states is assumed rather than guaranteed by the proposed GRU encoder. These issues raise concerns about whether the theoretical guarantees apply to the actual algorithm.
-
The experiments lack important ablation studies: since the joint observations recover the original D4RL observation, it is unclear whether GRU-based latent-state inference is necessary. Similarly, the paper does not compare MPIL-based refinement against standard NLL training under the same computational budget. Without these comparisons, it is difficult to determine which components are responsible for the reported improvements.
Questions for authors
-
What is the main advantage of MPIL-based refinement over standard NLL training with early stopping? Can the authors provide a comparison under the same computational budget?
-
How is the factorization in Assumption A.1 compatible with the GRU state update in eq. (1)? Does Theorem 5.1 apply to the implemented world model?
-
Since the joint observations recover the original D4RL observation, I'm curious why is GRU-based latent-state inference necessary. How does the method perform when directly using the joint observations?
5 comments
Response to Weakness 2
Weakness 2 Response
Eq. (1) describes the recursive GRU encoder used in the implementation, whereas Assumption A.1 specifies the idealized latent-state structure used in the theoretical analysis. Because the next latent state is computed from the next observation, the factorization in Assumption A.1 is not implied by the GRU update and requires an additional structural condition when the next observation remains conditionally stochastic. In the revised manuscript, we have made this distinction explicit and presented Theorem 5.1 as a conditional analysis under predictive sufficiency, kernel factorization, finite KL divergence, and coverage.
Predictive sufficiency is imposed as a representation assumption in our theoretical analysis: the latent state is assumed to preserve the information from the full history that is relevant to future prediction. The GRU reconstruction objective encourages the latent state to retain observation information but is not used to establish this assumption. Under the stated conditions, Theorem 5.1 provides a multi-agent performance analysis that incorporates errors in latent-state transitions, shared rewards, and local-observation generation into MPIL and connects these errors, together with offline-data coverage, to joint-policy regret. The implemented GRU world model is evaluated empirically through its predictive accuracy and downstream policy performance. Extending the analysis to approximately sufficient representations, for example through the approximate information-state framework, remains an important direction for future work.
For the implemented GRU–Gaussian model, these conditions are not satisfied: dependence of the next latent state on the next observation conflicts with the factorization when it is non-degenerate, and a degenerate latent transition is singular with respect to the Gaussian model. Thus Theorem 5.1 does not certify the implemented MuJoCo policies. Appendix A.3 now makes this limitation explicit.
Response to Weakness 3
Thank you for this suggestion. We have added a latent-state ablation and expanded the world-model refinement comparisons. We have also corrected our inaccurate statement that concatenating local observations recovers the original D4RL observation. In our implementation, the concatenated local observations differ from the original D4RL observation in both dimensionality and composition.
(1) We replaced the GRU latent state with the concatenation of the agents' current local observations while keeping the remaining world-model architecture, refinement procedure, decentralized actor inputs, and policy-learning settings unchanged. As shown in Table 2, the latent-state world model achieves lower mean observation-prediction RMSE in all 12 task–dataset settings. For example, the RMSE decreases from 0.1190 to 0.0270 on Hopper-random and from 0.4277 to 0.3311 on Walker2d-full-replay. The latent-state model also leads to higher mean downstream policy returns in every setting. Representative improvements include an increase from 22.39 to 43.32 on HalfCheetah-full-replay and from 51.65 to 63.57 on Walker2d-full-replay. These results show that integrating observation–action history through the GRU provides a more predictive representation than directly concatenating the current local observations, and that this benefit transfers to downstream policy learning.
(2) Proposition 4.1 establishes that MPIL and NLL are optimization-equivalent under its stated conditions; therefore, both variants are implemented through likelihood training. Section 6.3 and Fig. 3 compare the initial, refined, and Mixed world models and their downstream policies. The Mixed WM is the conventional one-shot NLL reference, trained on 0.5M samples from the mixed pool. Each refined model is initialized on 0.4M samples of one dataset type and refined with 0.1M additional samples from the mixed pool. Across the four Walker2d dataset types, the refined world models lead to higher downstream returns than the corresponding initial models. These results provide evidence of practical benefits from refinement and include a one-shot NLL reference with the same total number of samples. The Mixed WM uses a separately drawn set of 0.5M mixed samples, rather than the same final samples used by each refined model. This is a comparison at the same total sample count with different training datasets, so it does not isolate the effect of staged sample introduction. The current experiments do not establish a matched total gradient-update budget.
The latent-state ablation isolates the contribution of history-based inference. The refinement results show the performance of the staged procedure relative to the initial and one-shot Mixed references, subject to the data-composition and computation distinctions described above.
Response to Weakness 1
Thank you for raising this concern. We agree that the individual optimization components are largely standard. Under the conditions of Proposition 4.1, MPIL differs from NLL by a model-independent entropy term, so the two objectives have the same gradients and minimizers. We do not present MPIL as a new optimization algorithm. Its contribution is to give standard NLL training a policy-oriented interpretation: under the stated conditions, changes in NLL correspond to changes in posterior-averaged model discrepancy, which Theorem 5.1 relates to joint-policy regret under its own assumptions.
Our conditional analysis extends the perspective of SOReL's single-agent observed-state setting (Fellows et al., 2026) to latent-state cooperative multi-agent learning. MPIL accounts for errors in latent-state transitions, shared rewards, and local-observation generation, while the performance analysis considers joint actions and decentralized policies. SOReL measures model error in the state–action regions visited by a policy. Our MPIL instead measures model error on the latent-state–joint-action pairs in the offline data used to train the world model. To connect error on the offline data to error in the regions visited by the joint policy, we introduce a coverage condition and obtain Iπ~≤CID^N. The resulting conditional regret bound therefore depends on both offline-data model error and coverage of the regions visited by the joint policy. We have clarified the relationship to SOReL and the scope of this extension to latent-state multi-agent learning in the revised manuscript.
The experiments also test the practical value of staged refinement. Section 6.3 and Fig. 3 compare initial, refined, and Mixed world models on Walker2d. The Mixed WM is a conventional one-shot NLL reference at the same total sample budget of 0.5M transitions. Across the four Walker2d dataset types, the refined models lead to higher downstream policy returns than the corresponding initial models. These results support the practical benefit of refinement, while the shared likelihood objective remains unchanged. The Mixed WM uses a separately drawn set of 0.5M mixed samples, so the Mixed and refined models have the same total sample count but different training datasets. We therefore do not attribute this comparison exclusively to the order in which samples are introduced. The current experiments do not establish a matched total gradient-update budget. Our complete method also achieves higher mean returns than MOMA-PPO (Barde et al., 2024) in 10 of the 12 settings in Table 1; that comparison concerns complete methods rather than refinement alone.
References
• Paul Barde, Jakob N. Foerster, Derek Nowrouzezahrai, and Amy Zhang (2024). A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 141–150.
• Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel, Michael A. Osborne, and Jakob Nicolaus Foerster (2026). Fully Offline Reinforcement Learning. Reinforcement Learning Journal, volume 7.