acceptodds
Under review as a conference paper at ICLR 2027

Latent Joint-Action Modeling for Centralized Multi-Agent World Models

Abstract

World models have become an effective tool for improving sample efficiency in multi-agent reinforcement learning by enabling policy learning from imagined trajectories. However, centralized multi-agent world models must directly model complex joint actions, whose dimensionality and interaction structure grow with the number of agents. Moreover, different components of the joint action can be strongly correlated and need not contribute independently to future environment dynamics. Inspired by recent advances in latent action modeling in video world models and vision-language-action models, we propose LJAM, a Latent Joint-Action Modeling approach that represents the complex joint-action space through a structured latent effect space. Specifically, LJAM contextualizes each agent action with the current state and factorizes the resulting action tokens into a fixed set of collective, dynamics-relevant latent effects. We further introduce a predictive-error-aware confidence mechanism for weighting imagined policy updates. Experiments on multi-agent continuous-control benchmarks show that LJAM achieves state-of-the-art performance, while ablations demonstrate the importance of latent joint-action modeling and confidence-weighted updates.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, World Models, Model-Based Reinforcement Learning, Latent Action Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XzFjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LJAM, a latent joint-action modeling approach for centralized multi-agent world models. The method employs FiLM to construct state-conditioned action representations and uses Slot Attention to aggregate individual agent actions into a fixed number of latent joint-action effects for subsequent dynamics prediction. Additionally, the authors introduce a prediction-error-aware confidence weighting mechanism to reweight actor–critic updates during policy learning. Experiments on MAMuJoCo and Bi-DexHands demonstrate that the proposed method outperforms existing baselines on multiple tasks.

Critical strengths
  1. The paper addresses joint-action representation in multi-agent world models, which is a relevant and meaningful research problem.

  2. The method is evaluated on two continuous-control benchmarks, MAMuJoCo and Bi-DexHands, and achieves competitive performance across multiple tasks.

Critical weaknesses
  1. The proposed method mainly combines existing techniques, including FiLM, Slot Attention, and Cross-Attention. There is insufficient evidence that the learned slots capture distinct dynamical effects rather than simply serving as generic attention-aggregated features.

  2. The authors hypothesize that joint actions contain dynamics-relevant redundancy, but this assumption is neither explicitly quantified nor directly validated. Correlations among actions do not necessarily imply redundancy in their dynamical effects, and a fixed number of latent slots does not guarantee effective information compression.

  3. The paper devotes excessive space and equations to describing standard operations, resulting in a verbose and unfocused presentation that obscures the core technical contributions.

Questions for authors
  1. Confidence weighting only adjusts the actor–critic loss weights based on predicted errors. How does the method prevent errors from low-confidence transitions from propagating to other time steps through recursive return computation? Can the authors provide direct evidence that this mechanism effectively mitigates model bias?
3 comments
AuthorsOpenReview ↗

Additional Evidence for Latent Joint-Action Modeling and Confidence Weighting

Thank you for the detailed feedback. We have uploaded a revised PDF with additional slot-intervention and return-reliability analyses, and streamlined the method presentation.

W1: Existing components and the interpretation of slots

We agree that FiLM, Slot Attention, and Cross-Attention are established techniques. Our contribution concerns their use in a state-conditioned joint-action representation for centralized dynamics prediction, rather than new attention operators.

Contextualized tokens represent individual actions conditioned on state; joint-action effects aggregate across agents for transition prediction. We added interventions that zero one slot or replace it with its reference-action representation, holding real histories and candidate actions fixed.

For the checkpoint in Figure 5(a), either intervention increases full-state prediction error for either slot, with all corresponding 95% episode-bootstrap intervals above zero. In a common PCA basis fitted to real transition effects, the slots also contribute differently across transition directions (Figure 5(b)). Appendix Tables 5–7 report additional checkpoints and state-variable-group results.

The cross-attention checkpoint shows nearly identical intervention effects across its two tokens, whereas LJAM exhibits differentiated contributions. However, another LJAM checkpoint primarily shows a dominant-slot pattern, and the cross-attention comparison is not fully matched in architecture and training history. We therefore interpret these results as functional evidence, without claiming independent physical factors or fixed semantics across checkpoints.

W2: Evidence for latent joint-action modeling

Our central question is whether latent joint-action representations provide more effective conditioning for transition prediction than raw joint actions.

The state-change prediction probe obtains an MSE of 0.0688 with latent representations versus 0.0875 with raw joint actions, a 21.4% reduction; state-only inputs yield 0.1299. Latent similarities also more closely resemble real next-state effect similarities (Figure 8), supporting a representation better suited to transition prediction.

Figure 6(a) further compares slot aggregation with standard cross-attention, an MLP mapping, and direct action-token conditioning. Slot aggregation achieves higher policy returns on the evaluated task. The new interventions additionally establish that the transition model uses information from both slots.

Together, these results support the predictive and downstream value of latent joint-action modeling. A fixed slot count provides a fixed-size token interface; our empirical justification rests on predictive quality and functional contributions, rather than token-count reduction alone.

W3: Verbose presentation of standard operations

We have streamlined the method section by combining action encoding and FiLM parameter generation into inline definitions, consolidating the fusion equations into an overall transition-model interface, and shortening the decoder and reward/continuation descriptions. Repeated motivation was removed, while retaining slot-normalization axes, the multi-step dynamics objective, and confidence-weight construction and application.

Q1: Recursive error propagation and direct evidence

Confidence weights reweight actor–critic losses without changing return recursion. Later model errors can therefore still affect earlier targets; the mechanism does not explicitly block this propagation.

We added a paired finite-return diagnostic on two frozen Ant-8×1 checkpoints. Model and environment branches share 100 initial snapshots from 40 test episodes and identical recorded action sequences. We accumulate rewards and their respective continuation signals over 15 steps with discount 0.99 and no critic bootstrap. This evaluates accumulated reward/continuation errors, rather than reconstructing training lambda-return targets.

Training seedUniformPredictedShuffledReduction vs. Uniform, 95% CI
435533234.19324.09304.15602.39% [0.86%, 3.89%]
216855233.65613.60563.65541.38% [0.26%, 2.56%]

Values are absolute finite-return discrepancies. Predicted weights are normalized by their sum; Shuffled reports the exact expectation under within-depth permutations. Paired differences favor Predicted over both controls, with 95% episode-bootstrap intervals above zero. Equal averaging across depths retains reductions of 1.49% and 1.39%, with ESS/N of 0.948 and 0.952 (Appendix Tables 8–10).

The effects are modest and vary across depths. They show that the weights preferentially emphasize samples with smaller accumulated finite-return errors, supporting their intended reliability-selection role. They do not establish that recursive propagation is prevented or that bootstrapped targets or policy gradients are unbiased.

Regarding W3, the authors have not yet revised the manuscript to address this concern. The presentation remains overly verbose and unfocused, making it difficult to understand the proposed method and identify its core technical contributions. Therefore, my concern remains unresolved.

AuthorsOpenReview ↗

Further Manuscript Revisions to Address W3

Thank you for the follow-up. We understand that the previous revisions did not sufficiently clarify the method and its contributions. We have now uploaded a further revised manuscript with the following changes:

  1. Rewritten contribution statement (Introduction, p. 2). We replaced the abstract “new view” and “latent effect factorization” descriptions with explicit statements of the design: state-conditioned action tokens, competitive slot aggregation for dynamics prediction, and horizon-normalized error-based weighting for imagined updates. We also distinguish these designs from the established components they employ.

  2. Consolidated related work (Sections 2.1–2.2, pp. 2–3). We merged the previous multi-agent and sequential world-modeling subsections, shortened individual method descriptions while retaining all cited works, and consolidated the comparison with LJAM. The latent-action discussion has also been shortened and organized around its relationship to our modeling problem.

  3. Removed repeated motivation (Sections 3–4.1, pp. 3–4). We deleted repeated discussions of joint-action modeling from the end of the preliminaries and the beginning of Section 4.1. The presentation now proceeds from the method overview directly to state-conditioned action representations and slot aggregation.

  4. Reduced standard-operation exposition (Sections 4.1–4.2, pp. 4–5). Action encoding and FiLM parameter generation are now inline definitions accompanying one action-token equation. The separate fusion equations have been consolidated into an overall transition-model interface, and the observation decoder and reward/continuation interfaces are described inline. Relative to the earlier manuscript, numbered equations in the preliminaries and method sections have decreased from 17 to 12.

Importantly, Section 4.1 now explicitly explains the purpose of normalization across slots and subsequent renormalization across agents, including its distinction from standard cross-attention. This makes the design rationale visible alongside the equations.

These revisions aim to make the technical contributions identifiable without requiring the reader to work through repeated motivation or detailed descriptions of standard components. We would appreciate your reassessment of W3 based on the newly uploaded version.

Reviewer_qVzZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LJAM, a centralized multi-agent model-based RL algorithm that encodes joint action into KK continuous tokens using slot attention. An auxiliary error-prediction head is used to control the weight of policy update.

Critical strengths

The presentation is good.

Critical weaknesses
  • I am concerned about the novelty of this method. For the dynamics prediction of the model, it seems that it just replace the encoder to slot attention, which is quite similar to cross attention. I think the motivation to use slot attention should be explained more carefully, especially when KK is only 22 in practice.
  • The experiment diversity is not enough. The algorithm is only tested on MA-MuJoCo and Bi-Dexhand. Other benchmarks with discrete action space should also be tested (e.g, SMAC, Google Football).
  • The single agent baselines are missing. This is especially important on MA-MuJoCo, since on MA-MuJoCo, even single agent model-free approaches work better than Multi-agent model-based approaches. I know MA-MuJoCo is widely tested in this area, but if multi-agent model-based approaches cannot even outperform single-agent model-free baselines, the paradigm of "controlling each joint as an agent" itself might be questionable, which may further weaken the experimental result.
Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Motivation for Slot Aggregation and Additional Functional Analyses

We have uploaded a revised manuscript with additional slot-intervention analyses to clarify the motivation and functional role of slot aggregation. The main-text analysis in Section 5.2(i), Figure 5, evaluates individual-slot interventions and their contributions across transition directions. Appendix Tables 5–7 provide detailed state-variable-group results for two LJAM checkpoints and a standard cross-attention checkpoint. We summarize the relevant findings below.

Response on Slot Aggregation: Motivation and Functional Evidence

Thank you for raising this concern. We agree that Slot Attention and cross-attention share attention-based operations. Our contribution is a joint-action representation design for centralized dynamics prediction, rather than a new attention operator.

Motivation and the choice of K. Standard cross-attention normalizes attention over input tokens independently for each query. Our module follows Slot Attention by first normalizing assignments across slots, introducing competition for each action token, and then renormalizing across agents for aggregation. This encourages differentiated assignments without guaranteeing independent physical factors.

K denotes the number of aggregated action representations, rather than the number of agents or predefined dynamical factors. Figure 6(a) compares slot aggregation with standard cross-attention, an MLP mapping, and direct action-token conditioning, showing higher policy returns with the slot-based formulation. Figure 6(c) evaluates K = 1, 2, and 3; K = 2 performs slightly better overall and is used in the main experiments.

Additional functional evidence. Beyond attention visualization, we added interventions that zero one slot or replace it with its reference-action representation. Real input histories and candidate actions are held fixed, and predictions are evaluated against real environment transitions.

The following tables summarize the original state-variable-group results reported in Appendix Tables 6 and 7. Entries are signed increases in standardized MSE relative to the corresponding full model. Positive values indicate worse predictions after intervention. Full paired episode-bootstrap intervals are provided in the appendix.

LJAM-B: state-variable-group intervention results

State groupZero slot 1Zero slot 2Ref slot 1Ref slot 2
Full state+0.215+0.031+0.210+0.014
Torso height+1.510+0.059+1.185−0.185
Torso orientation+0.735+0.076+0.444−0.056
Joint position+0.892+0.119+0.936+0.072
Torso linear velocity+1.261+0.175+1.170+0.073
Torso angular velocity+0.643+0.093+0.733+0.041
Joint velocity+0.821+0.161+0.901+0.126

Standard cross-attention: state-variable-group intervention results

State groupZero token 1Zero token 2Ref token 1Ref token 2
Full state+0.015+0.015+0.093+0.093
Torso height+0.165+0.166+0.452+0.456
Torso orientation+0.029+0.028+0.165+0.164
Joint position+0.055+0.055+0.430+0.429
Torso linear velocity+0.140+0.140+0.503+0.503
Torso angular velocity+0.039+0.039+0.242+0.241
Joint velocity+0.051+0.051+0.441+0.439

The contact-force group is omitted here because its reported errors and intervention increments are zero.

The cross-attention checkpoint exhibits nearly identical intervention effects for its two tokens across these groups. In contrast, LJAM-B exhibits differentiated contributions, primarily with slot 1 dominating. We do not interpret this dominant-slot pattern alone as complementary specialization.

The main-text checkpoint, LJAM-A, provides additional directional evidence: in a common PCA basis fitted to real transition effects, slot 1 contributes more strongly to PCs 1, 2, 3, 7, and 8, whereas slot 2 contributes more strongly to PCs 5 and 6 under both intervention schemes (Figure 5(b)). Intervening on either slot also increases full-state prediction error, with all corresponding 95% intervals above zero (Figure 5(a)).

Together, these analyses support useful and potentially differentiated contributions even with two slots, beyond differences in attention weights. The cross-attention checkpoint is a descriptive comparison because its architecture and training history are not fully matched to LJAM; we therefore do not attribute these differences solely to competitive assignment or claim fixed physical semantics across checkpoints.

AuthorsOpenReview ↗

Additional SMAC Experiments and Single-Agent Baseline Comparisons

We have uploaded a revised PDF incorporating additional SMAC experiments and single-agent baseline comparisons. We have also clarified the differences in execution constraints, sample efficiency, and final performance. Please refer to the updated PDF for the corresponding results and analyses.

W2: Evaluation on discrete-action benchmarks

Thank you for this suggestion. We added experiments on four SMAC maps, involving separate combat units that coordinate movement and target selection through discrete actions.

MapLJAM stepsLJAMMARIEMAMBAMAPPOQMIX
3m100K100.00 (0.00)99.5 (0.4)87.7 (7.1)80.5 (12.8)54.4 (22.7)
8m100K100.00 (0.00)88.0 (3.9)65.0 (7.7)70.3 (19.5)69.5 (12.8)
25m200K53.33 (11.55)/27.0 (8.0)2.0 (2.0)8.0 (8.0)
27m_vs_30m400K3.33 (5.77)/0.00.00.0

Entries are win rates (%) as mean (standard deviation). LJAM uses three training seeds and sample standard deviations; baselines retain their reported statistics. Steps specifies LJAM's budget. “/” indicates unavailable results; standard deviations were not provided for the zero-valued baseline entries.

LJAM achieves 100% across all three seeds on 3m and 8m. The 25m and 27m_vs_30m maps contain 25 and 27 controlled agents, respectively, extending our evaluation beyond actuator-partitioned continuous control and eight-agent settings.

MARIE's training cost at these larger agent counts prevented us from completing its training within our available budget. LJAM completed training at both scales, providing practical evidence of computational scalability. Nevertheless, the low absolute win rate on 27m_vs_30m indicates that this task remains challenging. These results provide additional evidence for discrete-action cooperation and larger agent populations.

W3: Comparison with single-agent model-free methods

We agree that single-agent baselines provide important context for interpreting MAMuJoCo results. We therefore added the following comparison.

Task / LJAM partitionLJAMTD3DDPGSACA2CPPO
Ant / 8×16037.724372.441005.30≈37203485.41083.20
HalfCheetah / 3×26188.499636.953305.60≈104001829.91795.43
Walker2d / 3×23754.584682.821843.85≈37401091.63317.69

For compactness, entries report mean returns. TD3, DDPG, and PPO values are from the TD3 paper; SAC values are approximate values from the SAC paper as summarized by Tianshou; A2C values are from Tianshou. These published results provide context rather than a controlled comparison, since environment versions and evaluation protocols may differ.

We acknowledge that LJAM does not match the reported single-agent TD3 and SAC returns on HalfCheetah. However, the multi-agent formulation introduces additional challenges. Under CTDE, centralized information is available during training, but each actor selects its assigned actions using only its local observation during execution. A single-agent controller instead jointly selects all action components through one policy. Thus, even with the same underlying physical environment, MAMuJoCo introduces local-information constraints and cross-agent coordination requirements, making control more challenging than its centralized single-agent counterpart.

Our objective is therefore to improve learning under these decentralized execution constraints, rather than to establish that actuator partitioning outperforms centralized control. Under the same multi-agent formulation, Figures 3 and 4 show that LJAM achieves higher final mean returns than the evaluated MARL baselines and reaches strong returns with fewer real environment interactions. This supports its performance and sample-efficiency benefits within the MARL setting.

We have clarified these distinctions and acknowledged the remaining HalfCheetah performance gap. The additional SMAC experiments also demonstrate that our evaluation now extends beyond the actuator-partitioning paradigm.

Reviewer_qVzZOpenReview ↗

Response

Thank you for adding the discrete-action experiments. I recommand the authors to do a more thorough test on these tasks and include these results in their next version. However, I remain unconvinced by the novelty and motivation of slot attention. Its performance gains over alternative aggregation mechanisms appear marginal, and increasing the number of slots from two to three provides no further improvement. While this does not invalidate the design, the additional analyses do not sufficiently establish the need for slot-based factorization.

Regarding MAMuJoCo, the official setting is fully observable. So the claimed local-information constraints do not apply.

I therefore maintain my original score.

Reviewer_Lt7kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LJAM, a centralized multi-agent world model for cooperative MARL. The action of each agent is projected into an embedding, conditioned on global state through FiLM, and aggregated into a fixed set of latent joint-action effects using learnable queries. An auxiliary prediction-error head provides confidence weights for actor–critic updates on imagined trajectories. Experiments on MAMuJoCo and Bi-DexHands demonstrate LJAM’s effectiveness.

Critical strengths

The performance seems promising on the selected MAMuJoCo and Bi-DexHands tasks. Another strength is its explicit use of predicted model error to weight imagined actor–critic updates. This provides a practical mechanism for reducing the influence of less reliable model-generated transitions during policy learning.

Critical weaknesses
  • The proposed architecture combines established conditioning and token-aggregation mechanisms from VLA and VLM. For example, RT-1 [1] already combines FiLM conditioning with fixed-size token aggregation. Although RT-1 aggregates visual tokens for action prediction, whereas LJAM aggregates state-conditioned action tokens for dynamics prediction, this undermines the contribution of the proposed architecture.
  • The proposed joint-action effect seems a fancy name for contextualized action. The author should justify its design by designing more insightful experiment beyond the attention weight analysis, in order to interpret its mechanism.
  • Although the method show strong performance on the selected tasks, its applicability to generation multi-agent scnario remain unclear. MAMuJoCo includes up to eight agents, yet it is single-agent inherently. Tasks with more agents and complex interactions are required to provide the evidence for its generality as a multi-agent world model.

Reference:

  • [1] RT-1: Robotics Transformer for Real-World Control at Scale. RSS 2023.
Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response on Architectural Novelty, Slot Interpretation, and Multi-Agent Scalability

Thank you for the constructive feedback. We address the three concerns below and have revised the manuscript accordingly.

W1: Architectural novelty and the relationship to RT-1

We agree that FiLM and fixed-size token aggregation are established techniques, and we do not claim these building blocks themselves as novel. However, conditioning serves different modeling purposes in the two methods.

In RT-1, FiLM conditions visual features on language instructions for vision–language fusion. LJAM uses the shared state to generate modulation parameters that contextualize each agent's action before joint aggregation. This addresses the fact that the same action can have different dynamical consequences in different states.

Our design contribution is to combine state-conditioned individual action representations with joint-action aggregation for centralized dynamics prediction. Sharing the state encoding and modulation parameters provides a lightweight implementation. We have clarified this contribution and distinguished it from the established components employed.

W2: Interpretation beyond contextualized actions

Contextualized action tokens and latent joint-action effects represent two different levels in LJAM. The former incorporate state context into individual agents' actions; the latter aggregate information across agents and are learned through transition prediction. Thus, “effects” describes the aggregated representation's role in dynamics prediction, rather than simply renaming individual action tokens. It does not imply independent physical factors.

Our ablation compares slot aggregation with standard cross-attention, an MLP mapping, and directly supplying action tokens. Figure 6(a) shows higher policy returns with slot aggregation. The state-change prediction probe also obtains lower MSE with latent representations than with raw joint actions: 0.0688 versus 0.0875.

Following your suggestion, we added functional interventions beyond attention visualization. Holding real input histories and candidate actions fixed, we either zero an individual slot or replace it with its reference-action representation, then evaluate predictions against real environment transitions.

In the displayed checkpoint, both interventions increase full-state prediction error for either slot, with all corresponding 95% episode-bootstrap intervals above zero (Figure 5(a)). A common PCA basis fitted to real transition effects further reveals direction-dependent contributions across slots (Figure 5(b)). Additional checkpoints and state-variable-group results are provided in the appendix. These support predictive utility and differentiated functional contributions, without establishing fixed physical semantics across training runs.

W3: Larger and more general multi-agent settings

We acknowledge that actuator partitioning in MAMuJoCo alone does not establish generality. We therefore added SMAC experiments involving separate combat units with discrete actions and coordinated movement and target selection.

MapLJAM stepsLJAMMARIEMAMBAMAPPOQMIX
3m100K100.00 (0.00)99.5 (0.4)87.7 (7.1)80.5 (12.8)54.4 (22.7)
8m100K100.00 (0.00)88.0 (3.9)65.0 (7.7)70.3 (19.5)69.5 (12.8)
25m200K53.33 (11.55)/27.0 (8.0)2.0 (2.0)8.0 (8.0)
27m_vs_30m400K3.33 (5.77)/0.00.00.0

Entries are win rates (%) as mean (standard deviation). LJAM uses three training seeds and sample standard deviations; baselines retain their reported statistics. Steps specifies LJAM's budget. “/” indicates unavailable results; standard deviations were not provided for the zero-valued baseline entries.

The 25m and 27m_vs_30m maps contain 25 and 27 controlled agents, extending our evaluation beyond eight agents. LJAM achieves 100% across all seeds on 3m and 8m, and 53.33% on 25m.

MARIE's training cost at these larger agent counts prevented us from completing its training within our available budget. LJAM completed training at both scales, providing practical evidence of computational scalability. Nevertheless, its low absolute win rate on 27m_vs_30m shows that this map remains challenging. We present these results as additional evidence for discrete-action cooperation and larger agent populations, rather than unrestricted generality. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed below. Please refer to the updated PDF for the corresponding results and analyses.

I thank the authors for their explanations and the additional experiments. However, my concern regarding novelty remains. Although similar architectures have been applied in embodied AI, reapplying such an architecture to the multi-agent setting does not justify a sufficient technical contribution. I therefore maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.