acceptodds
Under review as a conference paper at ICLR 2027

Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

Abstract

Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, unstable training, weak coordination, and limited theoretical guarantees. In this paper, we propose the Consensus Multi-Agent Transformer (CMAT), a centralized framework that bridges cooperative MARL to a hierarchical single-agent reinforcement learning (SARL) formulation. CMAT treats all agents as a unified entity and employs a Transformer encoder to process the large joint observation space. To handle the extensive joint action space, we introduce a hierarchical decision-making mechanism in which a Transformer decoder autoregressively generates a high-level consensus vector, simulating the process by which agents reach agreement on their strategies in latent space. Conditioned on this consensus, all agents generate their actions simultaneously, enabling order-independent joint decision making and avoiding the sensitivity to action-generation order in conventional Multi-Agent Transformers (MAT). This factorization allows the joint policy to be optimized using single-agent PPO while preserving expressive coordination through the latent consensus. To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines. The code for this paper is available at an anonymous repository: https://anonymous.4open.science/r/CMAT-416A .

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZFmvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CMAT, a centralized framework that recasts fully observable cooperative multi-agent control as hierarchical single-agent reinforcement learning. Its Transformer encoder–decoder iteratively refines shared latent consensus for parallel action generation, optimized with PPO. Five-seed experiments across nine SMAC, Multi-Agent MuJoCo, and Google Research Football tasks report gains over baselines and examine consensus iterations, aggregation, and fine-tuning.

Critical strengths
  1. CMAT shifts autoregressive processing from individual actions to a shared latent consensus, then generates all agents’ actions in parallel. This removes the need to select or learn an agent action-generation order, offering a useful architectural alternative to MAT.
  2. The evaluation covers nine tasks across SMAC, Multi-Agent MuJoCo, and Google Research Football, with results reported over five random seeds. The tasks include both discrete and continuous control. These experiments provide preliminary evidence of practical value beyond a single benchmark and help assess the contribution of the proposed design choices.
Critical weaknesses
  1. The main-text expressiveness claim is inconsistent with the deterministic-consensus policy actually used. Section 3.1 argues that allowing c=μθ(O)c=\mu_\theta(O) to vary across observations and using a continuous consensus space enables CMAT to recover the full joint policy class represented by MAT's autoregressive factorization. However, the policy in Eq. (22) remains a single product distribution for any fixed observation OO and therefore cannot represent arbitrary within-state correlations among agents' actions. Appendix F.3 and Remark 7 explicitly acknowledge this limitation and clarify that the universal-expressiveness results apply only to the stochastic-consensus extension, directly contradicting the stronger claim in the main text.
  2. Training-budget comparability and architectural attribution remain insufficiently established. Both CMAT-finetune variants start from well-trained CMAT checkpoints, but Figure 3 resets their curves to zero, leaving equal-total-budget gains unverified (Sec. 4.2). Moreover, CMAT changes both MAT’s action-generation architecture and PPO clipping objective, replacing per-agent conditional ratios with a joint-action ratio. Although the m=0m=0 ablation supports decoder iteration within CMAT, the main comparison does not separate architectural and optimization effects or establish an advantage over a parameter-matched encoder-only controller. Attributing the cross-benchmark and fine-tuning gains specifically to latent consensus is therefore insufficiently supported.
Questions for authors
  1. Could you correct Eq. (1)’s conflation of joint and marginal distributions and derive Eq. (11) by marginalizing the deterministic latent rather than cancelling latent-policy factors? Does the code follow Algorithm 1, line 23, using the action-policy ratio with the respective current and behavior-policy consensus vectors and no high-level importance factor?
  2. How do the expressiveness claims in Section 3.1 and Remark 6 reconcile with Remark 7’s product-policy restriction at fixed OO? Please distinguish claims for deterministic consensus from those requiring stochastic consensus.
  3. What are the per-scenario checkpoint-training and fine-tuning interaction budgets? Do gains persist under equal cumulative budgets and against continued training of all modules from the same CMAT checkpoint for the same additional interactions?
  4. Since action heads also receive globally informed embeddings, can you supplement the m=0m=0 ablation and Appendix E.5 analysis with an encoder-only, parallel-head baseline matched for global inputs, joint PPO objective, parameters, and interaction budget? Can MAT comparisons under the same clipping objective isolate architectural gains?
  5. Minor presentation issue: Please fix the unresolved cross-references, including two instances of Table ?? and one of Fig. ?? on p. 4, and one of Fig. ?? on p. 7.
3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the detailed and constructive review. We have corrected the theoretical statements, clarified the comparison scope, and specified the evidence behind each point below.

Deterministic consensus and policy expressiveness

We have corrected the erroneous full-expressiveness claim: deterministic consensus gives a product policy at each fixed input, while a stochastic shared latent gives a mixture of products. The paper now presents CMAT as centralized SARL with a factored joint action trained by joint-action PPO.

The deterministic consensus coordinates the action-distribution parameters through shared global information. At a fixed input OO, its sampling law is

pθ(A∣O)=∏iπθi(ai∣O,μθ(O)).p_\theta(A\mid O)=\prod_i\pi^i_\theta(a_i\mid O,\mu_\theta(O)).

Changing the latent dimension or its value across observations preserves this factorization. The binary law supported equally on (0,0)(0,0) and (1,1)(1,1) illustrates the distinction: both marginals are fair Bernoulli distributions, whose independent product also assigns mass to the cross outcomes. A stochastic shared latent can produce the correlated law by mixing deterministic joint decisions. Appendix F analyzes that extension separately.

The contribution is shared latent refinement followed by parallel action generation, giving the heads a common computation path for state-dependent decisions.

Manuscript changes: The abstract, Introduction, Sections 2.1 and 3.1, Appendix E.1, and Appendix F.

Budget comparability and architectural attribution

We have corrected the attribution in Sections 4.2–4.3: Figure 3 compares complete controllers whose architectures, PPO objectives and training budgets differ.

Fine-tuning continues a trained policy. If checkpoint training uses T0T_0 interactions and continuation adds Δ\Delta, its cumulative cost is T0+ΔT_0+\Delta. The figure plots the continuation stage. Without full-model continuation from the same checkpoint for the same budget, the plotted gains reflect both further training and the chosen freeze scheme.

Clipping the joint ratio differs from aggregating separately clipped conditional ratios, so the CMAT/MAT comparison combines architectural and optimization effects. The m=0m=0 ablation retains the initial shared latent e0e^0; an encoder-only actor reads its contextualized token alone. A matched encoder-only comparison is not included in the current evaluation.

Manuscript changes: Sections 4.2–4.3, the Figure 3 caption, and Appendices D.3 and E.2.

Joint and marginal likelihoods in the implementation

We have separated the joint latent-action law from the action marginal in Eq. (1) and re-derived Eq. (11) from separately marginalized action likelihoods. The model recomputes the current consensus, and the trainer uses action log-probabilities without a latent-policy factor. The historical buffer's grouping of all agents from one transition on the summed axis cannot be confirmed from the supplied code.

For latent law qθ(dc∣O)q_\theta(dc\mid O), the joint law and action marginal are

p_\theta(A\mid O)=\int\prod_i\pi^i_\theta(a_i\mid O,c)\,q_\theta(dc\mid O).$$ Substituting $q_\theta=\delta_{\mu_\theta(O)}$ separately for the current and behavior policies gives $$R_t(\theta)=\exp\!\left[ \sum_i\log\pi^i_\theta(a_t^i\mid O_t,\mu_\theta(O_t)) -\sum_i\log\pi^i_{\bar\theta}(a_t^i\mid O_t,\mu_{\bar\theta}(O_t)) \right],$$ where $\bar\theta$ denotes behavior parameters. Both likelihoods evaluate the same sampled action using their respective encoders and consensuses. Saving the behavior log-probability at collection preserves the denominator; the numerator is recomputed for each update. For diagonal Gaussian heads, each head also sums over its action coordinates. The trainer subtracts the saved behavior log-probability and reduces its last tensor axis. To produce one joint-action ratio, that axis must contain all agents' likelihood contributions from one transition. Algorithm 1 and Appendix D.5 now specify this grouping. **Manuscript changes:** Section 2.1, Eq. (1); Section 3.3, Eqs. (10)–(11); Algorithm 1; Appendices E.3 and D.5.
AuthorsOpenReview ↗

Response Part 2

Deterministic versus stochastic consensus

We have made Section 3.1 and the appendix consistent: deterministic consensus gives a product distribution at each fixed input; stochastic consensus gives a mixture of products and can represent within-input action correlations.

For stochastic consensus,

pθ(A∣O)=∫∏iπθi(ai∣O,c) qθ(dc∣O).p_\theta(A\mid O)=\int\prod_i\pi^i_\theta(a_i\mid O,c)\,q_\theta(dc\mid O).

Sampling a common cc selects the heads' distributions together and can create action correlations after marginalization. Deterministic consensus selects one value of cc, leaving one product distribution.

For finite action spaces, Appendix F.2 gives a sufficient mixture construction with K≥M/max⁡i∣Ai∣K\ge M/\max_i|\mathcal A_i|, where M=∏i∣Ai∣M=\prod_i|\mathcal A_i|. Each mixture component fixes the actions of all agents except the one with the largest action space; that remaining head represents its conditional distribution. This construction represents any joint law using freely specified stochastic mixture factors. The evaluated CMAT uses deterministic consensus.

Manuscript changes: Section 3.1, Appendix E.1, Appendix F.1, and Appendix F.2, Proposition 2 and Eq. (33).

Interaction budgets and full-model continuation

Per-scenario checkpoint-training and fine-tuning counts are unavailable in the current records. Figure 3 contains no same-checkpoint, equal-budget full-model continuation, so it does not resolve whether the gains persist under that comparison. We have corrected the cumulative-budget accounting and the interpretation of these curves.

The fine-tuning curves start from trained CMAT checkpoints and display additional stage interactions, so their high initial returns reflect the already trained policies. If checkpoint training costs T0T_0 joint environment transitions and fine-tuning adds Δ\Delta, the total is T0+ΔT_0+\Delta. Section 3.3 specifies the trainable modules in each branch; Section 4.2 and the Figure 3 caption identify the checkpoint initialization and stage axis.

Manuscript changes: Section 3.3, Section 4.2, the Figure 3 caption, and Appendix D.3.

Matched encoder-only and MAT controls

The requested matched encoder-only and objective-matched MAT results are not part of the current evaluation. Matching PPO objectives would remove the objective difference; architectural attribution also requires matched inputs, capacity, critic and interaction budget.

The distinction from m=0m=0 is explicit: an encoder-only actor reads each globally contextualized token, while m=0m=0 also supplies the initial shared latent e0e^0. The existing ablation therefore measures decoder refinement within CMAT. Appendix E.5 now explains refinement as an additional shared computation path, replacing the earlier pooling/depth argument.

For action-factor ratios rir_i, the joint ratio is R=∏iriR=\prod_i r_i. Their clipping regions differ: with 0<ϵ<10<\epsilon<1 and r1=r2=1+ϵ/2r_1=r_2=1+\epsilon/2, each factor lies inside its interval, while R>1+ϵR>1+\epsilon. For positive advantage, joint clipping is already active even though neither factor is clipped.

Manuscript changes: Section 4.3, Appendix D.3, Appendix E.4, and Appendix E.5.

Cross-reference repairs

We have repaired the reported table and figure references. The payoff example is consistently cited as Figure 1 in Section 2.2, and the superseded payoff discussion in Section 3.3 has been removed.

Manuscript changes: Figure 1 and its references in Section 2.2; Section 3.3.

Reviewer_ZFmvOpenReview ↗

Response to the revision

Thank you for the revisions, which resolve some of my concerns.

  • I remain concerned about the fine-tuning comparisons. The per-scenario checkpoint-training and fine-tuning budgets are unavailable, and there is no full-model continuation from the same checkpoint for the same additional interactions. The curves do not establish gains at equal total budgets or benefits from freezing itself.

  • The within-CMAT ablations provide useful evidence about refinement under the same PPO objective. However, m=0m=0 still includes the initial shared latent, and the matched encoder-only and objective-matched MAT comparisons remain absent. The advantage over encoder-only remains untested, and the MAT comparison does not isolate architectural effects.

Given these gaps, I maintain my rating.

Reviewer_keoSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the "Consensus Multi-Agent Transformer" (CMAT), a centralized reinforcement learning framework that reformulates fully cooperative, fully observable multi-agent reinforcement learning (MARL) problems into single-agent problems with a decomposed joint action space. Unlike the autoregressive action generation approach employed by the "Multi-Agent Transformer" (MAT), CMAT incorporates an iterative "latent consensus mechanism". In this mechanism, a Transformer decoder continuously refines a shared representation, which then serves as a condition to guide all agents in generating their respective actions simultaneously. The policies produced by this framework are optimized using the PPO algorithm, supplemented by two fine-tuning strategies designed to refine the "consensus generation module" and the "action policies", respectively. Experiments conducted in SMAC, MA-MuJoCo, and GRF environments demonstrate that CMAT performs comparably to existing methods.

Critical strengths
  1. This paper focuses on the impact of the action generation order on the final decision performance in MAT and its variants, a problem with clear research significance. Through extensive analysis and argumentation, the authors propose a valuable alternative to autoregressive action generation in centralized multi-agent reinforcement learning (MARL). By circumventing explicit sequential action generation, this approach represents a promising direction for research.
  2. This paper proposes a hierarchical architecture that is both structurally clear and practically implementable, with its effectiveness validated through experiments across various types of multi-agent benchmark environments. The authors conducted experiments on multiple tasks in three benchmark environments: StarCraft II, Multi-Agent MuJoCo, and Google Research Football, covering both discrete and continuous action spaces. Furthermore, the study provides a comprehensive set of experimental benchmarks by comparing the proposed approach against methods such as MAT, PMAT, Triple-BERT, MAPPO, and HAPPO.
Critical weaknesses
  1. Insufficient ablation of core components and incomplete specification of the fine-tuning protocol. Figure 5 primarily examines the consensus aggregation strategy and the number of decoder iterations. Although the ablation includes m=0m=0, this setting retains the initial consensus derived from the globally contextualized encoder features rather than removing the consensus branch entirely. Consequently, the existing ablations do not sufficiently disentangle the contributions of globally shared representations and iterative consensus. A direct comparison with a parallel-action policy relying solely on the encoder features is still needed.
  2. There is a contradiction between the main text and the appendix. Section 3.1 claims that "over a continuous consensus space it reaches the full joint policy class that MAT obtains through the chain rule," whereas Remark 7 (in Appendix F.8) states that "the deterministic model cannot represent within-state correlations among agents’ actions that MAT’s autoregressive factorization can capture." Furthermore, there are issues with the writing; the main text contains several citation errors, resulting in placeholders such as "Table ??" and "Fig. ??".
Questions for authors
  1. CMAT-finetune performs particularly well in Figure 3, making its training protocol especially important for interpreting the reported results. Although the authors state that fine-tuning starts from a trained CMAT checkpoint, the exact timing and criterion for switching from joint training to fine-tuning, the total environment-interaction budget including the initial training phase, and the additional computational cost are not sufficiently specified. These omissions make it difficult to determine how much of the reported improvement stems from the consensus mechanism and staged optimization, rather than additional training budget.
  2. When the number of agents nn is very large (for example, in the massive-scale battles in SMAC, such as the 27m_vs_30m map), does setting the number of decoder iterations to m=nm=n incur an enormous computational overhead, thereby creating a significant computational bottleneck?
3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the careful and constructive review of the architecture, policy class, and training protocol. We have revised the manuscript and clarified the evidence behind each point below.

Component ablations and fine-tuning specification

We have corrected the ablation interpretation and specified the fine-tuning branches: m=0m=0 retains the initial consensus and tests decoder refinement within CMAT. A matched encoder-only result is not included in the current evaluation.

The ablation, moved into Section 4.3 as Figure 4, compares refinement length and history readout within CMAT. The history readout produces higher plotted returns on Ant, HalfCheetah and counterattack easy, whereas its differences from the last-latent variant are smaller on MMM2 and 3 vs 1 with keeper. Setting m=0m=0 gives c=e0c=e^0; an encoder-only actor would instead read its contextualized token without that additional shared latent.

Consensus supplies shared intermediate computation to the action heads. Refinement reuses decoder parameters, and history aggregation combines the intermediate states. Section 3.3 now specifies the separately frozen parameter blocks in each fine-tuning branch; Appendix D.3 accounts for their checkpoint-training cost.

Manuscript changes: Sections 3.1–3.3 and 4.3; Appendices D.2–D.3.

Policy expressiveness and cross-reference consistency

We have replaced the incorrect full-expressiveness claim with the statement that deterministic consensus gives a product policy at each fixed input. Stochastic consensus is treated separately as a mixture of products, and the unresolved cross-references have been repaired.

CMAT models centralized cooperative control as SARL with a factored joint action, trained with joint-action PPO. Its consensus coordinates the heads' distribution parameters through shared global information:

pθ(A∣O)=∏iπθi(ai∣O,μθ(O)).p_\theta(A\mid O)=\prod_i\pi^i_\theta(a_i\mid O,\mu_\theta(O)).

Increasing the deterministic latent's dimension or varying it across inputs changes those parameters while preserving the product law. For example, a joint distribution concentrated equally on (0,0)(0,0) and (1,1)(1,1) has Bernoulli marginals with parameter 1/21/2; multiplying these marginals gives probability 1/41/4 to every outcome, including the unwanted cross outcomes. A stochastic consensus supplies the shared sampling randomness needed for this correlated law, through the mixture of products analyzed in Appendix F.

The revised contribution is shared latent refinement followed by parallel action generation. The payoff example and its references now consistently resolve to Figure 1.

Manuscript changes: The abstract, Introduction, Sections 2.1–2.2 and 3.1, Appendix E.1, and Appendix F.

Fine-tuning schedule, cumulative budget, and attribution

We now specify the fine-tuning blocks and cumulative cost: each branch starts from a trained CMAT checkpoint, and Figure 3 plots only continuation interactions. The per-scenario switching steps, checkpoint-selection rules, total interaction counts and measured computation cannot be recovered from the current records.

Section 3.3 specifies the protocol. Both branches freeze the encoder and initial compressor. Consensus enhancement trains the decoder blocks, latent projector, history gate and critic head while fixing the action head. Action enhancement trains the action head and critic head with the consensus path fixed. The separate continuous-action standard-deviation parameter is frozen in both branches. A fixed action head still transmits input gradients to a trainable upstream consensus module.

If checkpoint training uses T0T_0 joint environment transitions and continuation adds Δ\Delta, the cumulative cost is T0+ΔT_0+\Delta. Computation includes checkpoint training, continuation and evaluation. Figure 3 reports the two frozen-block continuations; it does not include full-model continuation from the same checkpoint for the same additional budget. The plotted gains therefore combine further training and the chosen freeze scheme.

Manuscript changes: Section 3.3, Section 4.2, the Figure 3 caption, and Appendix D.3.

AuthorsOpenReview ↗

Response Part 2

Computational cost at large team sizes

Setting m=nm=n creates a scaling bottleneck for large teams: the current uncached implementation requires cubic attention work in the agent count and nn sequential latent passes. Appendix D.4 now derives this cost.

Each of the mm passes recomputes a length-mm latent sequence, using self-attention and cross-attention to nn encoder tokens. For width dd, LdL_d decoder blocks and a fixed attention-head count, its leading work per input is

O ⁣(Ld[m3d+m2nd+(m2+mn)d2]).O\!\left(L_d\left[m^3d+m^2nd+(m^2+mn)d^2\right]\right).

The cubic latent term comes from repeating length-mm self-attention, and the m2nm^2n term comes from repeated cross-attention. Projections and feed-forward layers give the remaining terms. The encoder adds O(Le[n2d+nd2])O(L_e[n^2d+nd^2]) for LeL_e layers. Thus parallel action output coexists with sequential latent computation, whose cost grows rapidly when refinement length tracks the agent count.

The iteration count can be chosen independently of nn. Figure 4 shows that the return response to refinement length varies by task, making mm a representation–computation tradeoff. Appendix D.4 now gives this implementation-specific cost analysis, and Section 4.3 presents m=nm=n as the evaluated setting.

Manuscript changes: Section 4.3; Appendix D.4, Eq. (15); Appendix E.5.

Thank you for the detailed response and manuscript revisions. I appreciate the authors' efforts to clarify the theoretical claims and training procedure. The additional complexity analysis is also helpful. However, some concerns remain. First, the lack of an encoder-only baseline makes it difficult to isolate the contribution of iterative consensus refinement. Second, the incomplete training records make it unclear whether the fine-tuning improvements come from the proposed method or additional training. Finally, the computational overhead may limit scalability to larger multi-agent systems. Overall, the response addresses several of my concerns, but some experimental limitations remain. I will maintain my original rating.

Reviewer_qTdtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on cooperative MARL, it builds fully centralized framework on MAT and proposes a method called CMAT. CMAT introduces a hierarchical decision making mechanism to hold a consensus vector, which enables order-independent joint action making.To evaluate the proposed method, we conduct experiments on benchmark tasks from StarCraft II, Multi-Agent MuJoCo, and Google Research Football. The results show that CMAT achieves superior performance over recent centralized solutions, sequential MARL methods, and conventional MARL baselines.

Critical strengths
  1. The paper addresses an interesting problem regarding action-generation-order dependence in centralized MARL.
  2. The experiments cover multiple benchmark families and include comparisons with several representative MARL methods.
Critical weaknesses
  1. The motivating game in Figure 1 is not convincingly. When MAT observes (B, A) with a negative reward, it may discourage Agent 1 from choosing B, but it also discourages Agent 2 from choosing A conditioned on Agent 1 choosing B, potentially promoting better coordination. Could you present stronger theoretical or empirical evidence to support your motivation?

  2. Could you explain what benefit iterative consensus provides for order-independent decision? The same question for the actor compressor and critic compressor.

  3. Experiment is not strong enough. Please check the questions below.

4.There are many figure and table reference errors, for example, in lines 183, 201, and 213.

Questions for authors

Motivation and Method

  1. The authors use a game to motivate the method, but I have questions about this game. When MAT observes (B, A) with a reward of −100, it does not only decrease the probability that Agent 1 chooses B. It might also decrease the probability that Agent 2 chooses A given that Agent 1 chose B. Since this is a cooperative setting, Agent 1's action will influence Agent 2's action. Do you have stronger evidence that order independence really matters in MARL?

  2. I notice that the consensus is deterministic. There is no separate stochastic process that selects between a suboptimal consensus and the optimal consensus. Once the model reaches a suboptimal consensus, it may be difficult to explore other possibilities.

  3. I cannot understand which architectural design choices enable order independence. Could you explain this? Whether iterative consensus is necessary for this property?

Experiment

  1. From my understanding, CMAT uses centralized training and centralized execution (CTCE), while MAPPO and HAPPO use centralized training and decentralized execution (CTDE). I do not think MAPPO and HAPPO are meaningful baselines.

  2. In Figure 3, seven of the nine CMAT variants do not show increasing curves. Could you explain this?

  3. Although Figure 3 shows that CMAT outperforms the baselines, the experiments do not prove that the performance improvements are due to order independence.

  4. For the environments you chose, why does order independence matter in these environments?

  5. You claim that the shared consensus representation helps agents coordinate. Intuitively, more iterations should lead to better performance, but the results in Figure 5 do not show this trend.

  6. The paper does not report efficiency results. It seems that CMAT has more components than MAT. I would like to know about its training efficiency.

3 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the careful and constructive review. We have revised the manuscript and clarified the evidence behind each point below.

Order effects and conditional learning signals in Figure 1

We have added the leading-agent gradient and the follower's conditional gradient after prefix BB: the follower improves, and the leading policy controls how often that response is sampled.

Let p=Pr⁡(a1=B)p=\Pr(a_1=B), q=Pr⁡(a2=B∣a1=B)q=\Pr(a_2=B\mid a_1=B) and u=Pr⁡(a2=B∣a1=A)u=\Pr(a_2=B\mid a_1=A). For separate tabular Bernoulli logits, J=(1−p)VA+pVBJ=(1-p)V_A+pV_B, with VA=1−uV_A=1-u, VB=200q−100V_B=200q-100, and

∂zpJ=p(1−p)(VB−VA),∂zqJ=200pq(1−q).\partial_{z_p}J=p(1-p)(V_B-V_A),\qquad \partial_{z_q}J=200pq(1-q).

Here 200=100−(−100)200=100-(-100) is the follower's payoff difference after prefix BB.

When VB<VAV_B<V_A, pp falls while qq improves. Lower pp means fewer prefix samples and, at fixed qq, a smaller follower gradient. Sufficient improvement in qq reverses the leading update. The leading policy thus controls access to the samples needed to make BB favorable.

Manuscript changes: Section 2.2 and Figure 1; Appendix E.4.

The role of iterative consensus and the two compressors

Parallel heads remove the action-generation order; iteration and the compressors compute shared context.

The Critic-Compressor supplies e0e^0 and the team-value input. The decoder repeatedly refines the latent with shared weights; the Actor-Compressor combines the intermediate latents into the context used by all heads. Encoder-only parallel policies can share the same generation and symmetry properties; iteration supplies additional shared computation. The m=0m=0 ablation retains c=e0c=e^0, testing decoder refinement within CMAT. Changing the initial compressor also changes value learning.

Manuscript changes: Sections 3.1–3.2 and 4.3; Appendices D.2–D.3, E.2 and E.5.

What the current experiments support

We have added theoretical analysis and clarified the existing experiments; no new benchmark runs are included.

The revised Figure 3 discussion includes checkpoint cost and distinguishes system performance from architectural attribution. Figure 4 examines refinement depth and history readout within CMAT. The added symmetry proof and cost analysis explain the architecture's ordering properties and computational requirements.

Manuscript changes: Section 4 and the Figure 3 caption; Appendices D.3–D.4 and E.2.

Figure and table references

We have repaired the identified figure and table references. Section 2.2 now consistently cites the payoff example as Figure 1, and the superseded payoff argument in Section 3.3 has been removed.

Manuscript changes: Section 2.2, Figure 1 and Section 3.3.

Conditional updates and the importance of action order

We have added a direct order comparison: from the same initial joint law, the probability of the lower-return equilibrium initially increases in one order and decreases in the other.

Initialize independent tabular Bernoulli logits to the uniform joint law and update all logits simultaneously by expected-gradient ascent with common step size η\eta. Appendix E.4 derives

=1/32\ \text{for }1\!\to\!2,\qquad -49/32\ \text{for }2\!\to\!1.$$ The signs describe opposite initial changes in the probability of $(A,A)$. With a zero baseline, a negative $(B,A)$ sample improves the follower's response after $B$; the leading policy controls how often that response is sampled. The gradients isolate a local learning effect: either unrestricted autoregressive order can represent any joint law, yet the orders induce different updates. CMAT removes action-prefix conditioning through shared latent computation and parallel sampling. **Manuscript changes:** Section 2.2 and Figure 1; Appendix E.4; Sections 4.2–4.3. ## Exploration with deterministic consensus CMAT explores through stochastic action heads and trains its deterministic consensus through joint-action PPO; it does not sample latent strategies independently. At fixed input, the heads sample independently. Their log-probabilities transmit reward-weighted gradients through the heads to the consensus and encoder. Shared parameters couple updates across inputs and latent values. Escape from a suboptimal policy depends on action exploration and optimization. Appendix F separately analyzes stochastic consensus, which can provide correlated exploration through shared latent randomness. **Manuscript changes:** Sections 2.1, 3.1 and 3.3; Appendices E.1, E.3, E.5 and F.1.
AuthorsOpenReview ↗

Response Part 2

Architectural sources of order independence

Parallel heads remove the action-generation sequence; shared token processing and invariant aggregation give input-permutation equivariance. Iteration is unnecessary for either.

Proposition 1 uses shared token layers without agent-slot positions, invariant pooling and latent cross-attention, and shared or identity-consistent heads. The consensus/value are invariant and action-distribution parameters are equivariant. Identities, roles, masks, target coordinates and dispatch move consistently with entities; latent positions index computation steps. Encoder-only parallel policies can satisfy the same conditions.

Manuscript changes: Section 3.2; Appendix E.2, Proposition 1 and its proof.

The role of MAPPO and HAPPO as baselines

MAPPO/HAPPO are CTDE system references; centralized MAT-family baselines are closer comparisons.

CTDE uses centralized critics and local actor inputs at execution, so comparisons with centralized execution also reflect information access. Globally informed encoder-only policies and objective-matched MAT would provide architectural controls. The MAPPO/HAPPO names are those of the original results; their actor-input configurations cannot be fully reconstructed from the available materials.

Manuscript changes: Section 4.1; Appendices D.1 and D.3.

Flat curves in Figure 3

We have clarified that the fine-tuning curves start from trained checkpoints and show only continuation interactions, which accounts for their high starting returns. The available records do not identify a cause for each flat trace.

Checkpoint cost T0T_0 plus continuation Δ\Delta gives cumulative cost T0+ΔT_0+\Delta. A flat trace records stable return over the displayed interval; distinguishing convergence, a learning plateau and unsuccessful learning requires the corresponding training and evaluation records.

Manuscript changes: Section 4.2, the Figure 3 caption and Appendix D.3.

What Figure 3 can establish about return gains

We have removed the attribution of Figure 3's gains specifically to order independence; it compares complete controllers.

CMAT/MAT differ in architecture and PPO: clipping a product of ratios differs from aggregating separately clipped factors. The evidence for the ordering properties comes from the parallel action-generation design and Proposition 1, which proves token-permutation equivariance under consistent entity semantics.

Manuscript changes: Sections 4.2–4.3 and the conclusion; Appendices D.3, E.2 and E.4.

Why order-independent execution matters in the environments

These environments execute joint actions at each step; parallel generation removes a serialization choice internal to the controller.

The components control SMAC units, MuJoCo actuator groups and football players. Each head reads shared context without waiting for another sampled action. Consistent entity relisting preserves the physical action distribution when identity, mask, target-coordinate and output mappings are aligned.

Manuscript changes: Section 4.2; Appendices D.4 and E.2.

Why more iterations do not guarantee higher returns

More iterations do not imply higher return: each depth is trained separately, rather than taking extra steps of a return-ascent solver.

Changing mm changes the unrolled decoder, readout history and gradient paths. Figure 4 shows task-dependent preferred depths. We retain m=nm=n as the evaluated setting; larger mm adds sequential work.

Manuscript changes: Section 4.3, Appendix D.2 and Figure 4; Appendix E.5.

Training computational cost

We have added a cost analysis: the uncached decoder retains sequential passes and cubic attention work when m=nm=n. Matched timing and peak-memory measurements against MAT are unavailable.

For nn agents, mm steps, width dd and LdL_d decoder blocks, the per-input decoder cost in Eq. (15) is

O ⁣(Ld[m3d+m2nd+(m2+mn)d2]).O\!\left(L_d[m^3d+m^2nd+(m^2+mn)d^2]\right).

The first two terms come from repeated self- and cross-attention; the remaining terms are projection/feed-forward work. The encoder adds O(Le[n2d+nd2])O(L_e[n^2d+nd^2]) for LeL_e layers; batch cost multiplies by batch size. Shared weights still require training activations across passes. Freezing saves parameter-gradient and optimizer work; forward computation and any upstream input gradients remain. Fine-tuning cost includes checkpoint training.

Manuscript changes: Section 4.3; Appendix D.4, Eq. (15), and its computational discussion.

Thank you for the detailed rebuttal and revisions. I appreciate the additional theoretical analysis and the clearer explanations of the architectural design. However, my main concerns remain unresolved.

  1. The experiments still do not establish whether removing action-order dependence contributes to the performance improvements. In particular, CMAT and MAT differ in both architecture and PPO objectives, making it difficult to isolate the contribution of order independence.

  2. I remain concerned about the comparison with MAPPO and HAPPO. As CTDE methods, their actors have access to local observations during execution, while CMAT uses global information under CTCE. Although MAPPO and HAPPO can serve as system-level references, their comparisons with CMAT cannot isolate the benefits of the proposed architecture due to the differences in execution-time information access.

  3. The revised manuscript states that order independence comes from parallel action generation. Then what does the consensus module add? The m = 0 ablation still keeps the shared latent e⁰, so it is not an encoder-only policy. A comparison with an encoder-only parallel-action policy is needed to show the benefit of the consensus module.

Therefore, I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.