- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CMAT, a centralized framework that recasts fully observable cooperative multi-agent control as hierarchical single-agent reinforcement learning. Its Transformer encoder–decoder iteratively refines shared latent consensus for parallel action generation, optimized with PPO. Five-seed experiments across nine SMAC, Multi-Agent MuJoCo, and Google Research Football tasks report gains over baselines and examine consensus iterations, aggregation, and fine-tuning.
Critical strengths
- CMAT shifts autoregressive processing from individual actions to a shared latent consensus, then generates all agents’ actions in parallel. This removes the need to select or learn an agent action-generation order, offering a useful architectural alternative to MAT.
- The evaluation covers nine tasks across SMAC, Multi-Agent MuJoCo, and Google Research Football, with results reported over five random seeds. The tasks include both discrete and continuous control. These experiments provide preliminary evidence of practical value beyond a single benchmark and help assess the contribution of the proposed design choices.
Critical weaknesses
- The main-text expressiveness claim is inconsistent with the deterministic-consensus policy actually used. Section 3.1 argues that allowing to vary across observations and using a continuous consensus space enables CMAT to recover the full joint policy class represented by MAT's autoregressive factorization. However, the policy in Eq. (22) remains a single product distribution for any fixed observation and therefore cannot represent arbitrary within-state correlations among agents' actions. Appendix F.3 and Remark 7 explicitly acknowledge this limitation and clarify that the universal-expressiveness results apply only to the stochastic-consensus extension, directly contradicting the stronger claim in the main text.
- Training-budget comparability and architectural attribution remain insufficiently established. Both CMAT-finetune variants start from well-trained CMAT checkpoints, but Figure 3 resets their curves to zero, leaving equal-total-budget gains unverified (Sec. 4.2). Moreover, CMAT changes both MAT’s action-generation architecture and PPO clipping objective, replacing per-agent conditional ratios with a joint-action ratio. Although the ablation supports decoder iteration within CMAT, the main comparison does not separate architectural and optimization effects or establish an advantage over a parameter-matched encoder-only controller. Attributing the cross-benchmark and fine-tuning gains specifically to latent consensus is therefore insufficiently supported.
Questions for authors
- Could you correct Eq. (1)’s conflation of joint and marginal distributions and derive Eq. (11) by marginalizing the deterministic latent rather than cancelling latent-policy factors? Does the code follow Algorithm 1, line 23, using the action-policy ratio with the respective current and behavior-policy consensus vectors and no high-level importance factor?
- How do the expressiveness claims in Section 3.1 and Remark 6 reconcile with Remark 7’s product-policy restriction at fixed ? Please distinguish claims for deterministic consensus from those requiring stochastic consensus.
- What are the per-scenario checkpoint-training and fine-tuning interaction budgets? Do gains persist under equal cumulative budgets and against continued training of all modules from the same CMAT checkpoint for the same additional interactions?
- Since action heads also receive globally informed embeddings, can you supplement the ablation and Appendix E.5 analysis with an encoder-only, parallel-head baseline matched for global inputs, joint PPO objective, parameters, and interaction budget? Can MAT comparisons under the same clipping objective isolate architectural gains?
- Minor presentation issue: Please fix the unresolved cross-references, including two instances of Table ?? and one of Fig. ?? on p. 4, and one of Fig. ?? on p. 7.
3 comments
Response Part 2
Deterministic versus stochastic consensus
We have made Section 3.1 and the appendix consistent: deterministic consensus gives a product distribution at each fixed input; stochastic consensus gives a mixture of products and can represent within-input action correlations.
For stochastic consensus,
Sampling a common selects the heads' distributions together and can create action correlations after marginalization. Deterministic consensus selects one value of , leaving one product distribution.
For finite action spaces, Appendix F.2 gives a sufficient mixture construction with , where . Each mixture component fixes the actions of all agents except the one with the largest action space; that remaining head represents its conditional distribution. This construction represents any joint law using freely specified stochastic mixture factors. The evaluated CMAT uses deterministic consensus.
Manuscript changes: Section 3.1, Appendix E.1, Appendix F.1, and Appendix F.2, Proposition 2 and Eq. (33).
Interaction budgets and full-model continuation
Per-scenario checkpoint-training and fine-tuning counts are unavailable in the current records. Figure 3 contains no same-checkpoint, equal-budget full-model continuation, so it does not resolve whether the gains persist under that comparison. We have corrected the cumulative-budget accounting and the interpretation of these curves.
The fine-tuning curves start from trained CMAT checkpoints and display additional stage interactions, so their high initial returns reflect the already trained policies. If checkpoint training costs joint environment transitions and fine-tuning adds , the total is . Section 3.3 specifies the trainable modules in each branch; Section 4.2 and the Figure 3 caption identify the checkpoint initialization and stage axis.
Manuscript changes: Section 3.3, Section 4.2, the Figure 3 caption, and Appendix D.3.
Matched encoder-only and MAT controls
The requested matched encoder-only and objective-matched MAT results are not part of the current evaluation. Matching PPO objectives would remove the objective difference; architectural attribution also requires matched inputs, capacity, critic and interaction budget.
The distinction from is explicit: an encoder-only actor reads each globally contextualized token, while also supplies the initial shared latent . The existing ablation therefore measures decoder refinement within CMAT. Appendix E.5 now explains refinement as an additional shared computation path, replacing the earlier pooling/depth argument.
For action-factor ratios , the joint ratio is . Their clipping regions differ: with and , each factor lies inside its interval, while . For positive advantage, joint clipping is already active even though neither factor is clipped.
Manuscript changes: Section 4.3, Appendix D.3, Appendix E.4, and Appendix E.5.
Cross-reference repairs
We have repaired the reported table and figure references. The payoff example is consistently cited as Figure 1 in Section 2.2, and the superseded payoff discussion in Section 3.3 has been removed.
Manuscript changes: Figure 1 and its references in Section 2.2; Section 3.3.
Response to the revision
Thank you for the revisions, which resolve some of my concerns.
-
I remain concerned about the fine-tuning comparisons. The per-scenario checkpoint-training and fine-tuning budgets are unavailable, and there is no full-model continuation from the same checkpoint for the same additional interactions. The curves do not establish gains at equal total budgets or benefits from freezing itself.
-
The within-CMAT ablations provide useful evidence about refinement under the same PPO objective. However, still includes the initial shared latent, and the matched encoder-only and objective-matched MAT comparisons remain absent. The advantage over encoder-only remains untested, and the MAT comparison does not isolate architectural effects.
Given these gaps, I maintain my rating.
Response Part 1
Thank you for the detailed and constructive review. We have corrected the theoretical statements, clarified the comparison scope, and specified the evidence behind each point below.
Deterministic consensus and policy expressiveness
We have corrected the erroneous full-expressiveness claim: deterministic consensus gives a product policy at each fixed input, while a stochastic shared latent gives a mixture of products. The paper now presents CMAT as centralized SARL with a factored joint action trained by joint-action PPO.
The deterministic consensus coordinates the action-distribution parameters through shared global information. At a fixed input O, its sampling law is
pθ(A∣O)=∏iπθi(ai∣O,μθ(O)).
Changing the latent dimension or its value across observations preserves this factorization. The binary law supported equally on (0,0) and (1,1) illustrates the distinction: both marginals are fair Bernoulli distributions, whose independent product also assigns mass to the cross outcomes. A stochastic shared latent can produce the correlated law by mixing deterministic joint decisions. Appendix F analyzes that extension separately.
The contribution is shared latent refinement followed by parallel action generation, giving the heads a common computation path for state-dependent decisions.
Manuscript changes: The abstract, Introduction, Sections 2.1 and 3.1, Appendix E.1, and Appendix F.
Budget comparability and architectural attribution
We have corrected the attribution in Sections 4.2–4.3: Figure 3 compares complete controllers whose architectures, PPO objectives and training budgets differ.
Fine-tuning continues a trained policy. If checkpoint training uses T0 interactions and continuation adds Δ, its cumulative cost is T0+Δ. The figure plots the continuation stage. Without full-model continuation from the same checkpoint for the same budget, the plotted gains reflect both further training and the chosen freeze scheme.
Clipping the joint ratio differs from aggregating separately clipped conditional ratios, so the CMAT/MAT comparison combines architectural and optimization effects. The m=0 ablation retains the initial shared latent e0; an encoder-only actor reads its contextualized token alone. A matched encoder-only comparison is not included in the current evaluation.
Manuscript changes: Sections 4.2–4.3, the Figure 3 caption, and Appendices D.3 and E.2.
Joint and marginal likelihoods in the implementation
We have separated the joint latent-action law from the action marginal in Eq. (1) and re-derived Eq. (11) from separately marginalized action likelihoods. The model recomputes the current consensus, and the trainer uses action log-probabilities without a latent-policy factor. The historical buffer's grouping of all agents from one transition on the summed axis cannot be confirmed from the supplied code.
For latent law qθ(dc∣O), the joint law and action marginal are
p_\theta(A\mid O)=\int\prod_i\pi^i_\theta(a_i\mid O,c)\,q_\theta(dc\mid O).$$ Substituting $q_\theta=\delta_{\mu_\theta(O)}$ separately for the current and behavior policies gives $$R_t(\theta)=\exp\!\left[ \sum_i\log\pi^i_\theta(a_t^i\mid O_t,\mu_\theta(O_t)) -\sum_i\log\pi^i_{\bar\theta}(a_t^i\mid O_t,\mu_{\bar\theta}(O_t)) \right],$$ where $\bar\theta$ denotes behavior parameters. Both likelihoods evaluate the same sampled action using their respective encoders and consensuses. Saving the behavior log-probability at collection preserves the denominator; the numerator is recomputed for each update. For diagonal Gaussian heads, each head also sums over its action coordinates. The trainer subtracts the saved behavior log-probability and reduces its last tensor axis. To produce one joint-action ratio, that axis must contain all agents' likelihood contributions from one transition. Algorithm 1 and Appendix D.5 now specify this grouping. **Manuscript changes:** Section 2.1, Eq. (1); Section 3.3, Eqs. (10)–(11); Algorithm 1; Appendices E.3 and D.5.