acceptodds
Under review as a conference paper at ICLR 2027

Coordination from Within: Decentralized Sequence Modeling for Multi-Agent Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning (MARL) often requires agents to account for one another's behavior from partial observations. Under centralized training with centralized execution (CTCE), sequence modeling can capture inter-agent interactions by jointly processing team observations. Extending this approach to centralized training with decentralized execution (CTDE) is challenging because each actor must select actions from its own local history. We propose Decentralized Sequence Modeling (DSM), equipping each actor with a persistent sequence of coordination tokens that represents cooperative behavior in latent space. Our Coordination-Token Actor uses a retention layer to process these tokens together with the current local input, producing a decision representation and updating the tokens for later decisions. We introduce Coordination-Token Distillation to supervise the updated tokens with a centralized teacher's representations of joint observations. Used only during training, the teacher also guides the local policy through its action distributions. Across 44 tasks in six benchmarks, DSM achieves the highest aggregate performance among the CTDE baselines on all six and matches or exceeds the best CTCE baseline on five benchmarks. These gains come with about 68% fewer deployed parameters than the reference GRU actor.

Primary area:
reinforcement learning
Keywords:
cooperative multi-agent reinforcement learning, decentralized sequence modeling, coordination tokens, centralized training with decentralized execution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SDjHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Decentralized Sequence Modeling (DSM), combining a retention-based Coordination-Token Actor (CTA) with Coordination-Token Distillation (CTD). Building on MAGPO’s policy-guidance framework, DSM supervises persistent local memory using projected centralized teacher features, while retaining fully decentralized execution. The empirical results are promising.

Critical strengths
  • A meaningful extension from action guidance to recurrent-memory supervision.
  • Broad empirical evaluation with substantial improvements on challenging tasks.
Critical weaknesses
  1. It is unclear whether CTD preserves useful coordination information rather than merely making the representations easier to match.

In Eq. (8), both the actor tokens and the teacher-side projector are optimized. The alignment loss can therefore decrease either because the actor learns useful teacher information or because the projector discards information that is difficult to match. The variance penalty prevents constant outputs but does not establish that the remaining variation is relevant to coordination. This does not imply that the method degenerates in practice; rather, the paper needs more direct evidence about what information the tokens retain and use for decisions.

  1. The specific benefit of supervising persistent memory is not sufficiently isolated.

The GRU+CTD and no-CTD ablations provide useful evidence for the combined design. However, they do not directly test whether supervising the updated memory tokens is more effective than applying comparable representation supervision elsewhere in the same actor. A comparison between supervising CtC_t and the decision representation ztz_t, with comparable projector capacity and loss scaling, would clarify whether the gains specifically support memory supervision or primarily reflect the benefits of an auxiliary representation objective.

Questions for authors
  1. Why is the agent-specific encoder feature o^ti\hat o_t^i an appropriate target for future local memory?

  2. Do the multi-token gains persist after controlling for recurrent-state size and key–value write magnitude?

  3. What are the runtime memory, inference latency, and training cost relative to MAGPO?

  4. Could you specify the GRU+CTD and QMIX distillation interfaces and the final per-task hyperparameters?

3 comments
AuthorsOpenReview ↗

Response to Reviewer SDjH (Part 1/3)

Thank you for distinguishing action guidance from supervision of memory used in later decisions. We reran Sable with the updated MPE observations and corrected DSM’s observations on con-15x15x23a (Table 1, footnotes 1–2, p. 6).

W1: Could the projector make the targets easier to match while discarding information that matters for coordination?

A projection can discard teacher information. Eq. (8) detaches the teacher feature before projection, so alignment updates the actor and projector. Eq. (9) penalizes low variance on both sides. We tested whether the variation retained in the targets matters for control.

We held the architecture, sampled targets and full CTD objective fixed and shuffled targets across timesteps within an episode fragment or across agents at the same timestep. On all four tasks, matched CTD finishes above the no-CTD actor and both shuffles finish below it. On RWARE large-4ag-hard, both shuffles stay near zero. The gain depends on assigning each update the context of its own agent and timestep. Fig. 7, p. 8

On SMAX 6h_vs_8z, the four slots overlap in the initial t-SNE projection and occupy distinct regions after training. Slot identity is thus visible in the learned representation. In Fig. R2(c–f), each point is an independently trained run evaluated at the same training step as the other runs on that task. Smaller held-out alignment errors accompany higher decentralized returns across runs. The Pearson correlations are −0.995-0.995 on CoordSum 8x15, −0.942-0.942 and −0.922-0.922 on RWARE large-8ag-hard and large-4ag-hard, and −0.965-0.965 on SMAX 10m_vs_11m. Together with the shuffle control, these observations support the usefulness of the projected context for local decisions. Fig. R2, p. 9

Table 3 (Appendix A.4, p. 18) covers all 44 tasks. Entries are final-policy suite IQMs ± bootstrap SDs, using normalized returns for the first five suites and win-rate fractions for SMAX.

SuiteMAGPONo CTDK=1DSM
CoordSum0.89 ± 0.0040.91 ± 0.0090.95 ± 0.0060.97 ± 0.003
LBF1.00 ± 0.0001.00 ± 0.0001.00 ± 0.0001.00 ± 0.000
Connector0.94 ± 0.0010.94 ± 0.0020.94 ± 0.0030.95 ± 0.001
MPE0.84 ± 0.0040.85 ± 0.0040.85 ± 0.0030.86 ± 0.002
RWARE0.64 ± 0.0060.68 ± 0.0100.72 ± 0.0060.76 ± 0.005
SMAX0.95 ± 0.0030.96 ± 0.0060.97 ± 0.0040.98 ± 0.001

On CoordSum and RWARE, removing CTD still leaves the token actor ahead of MAGPO. Supervising the same memory raises the IQM from 0.91 to 0.97 and from 0.68 to 0.76, respectively.

Q1: Why is the teacher’s agent-specific encoder feature an appropriate target for memory that will be used in future local decisions?

The teacher encodes joint observations into an agent-specific feature o^ti\hat{o}^i_t before decoding any current actions. CTD projects that agent’s team context onto its updated tokens Cti,1:KC_t^{i,1:K}. At the next step, the actor reads those tokens with its new local observation, previous action and retention state to form zt+1iz_{t+1}^i (Secs. 3.1–3.2).

For a fixed teacher and projector, the predictor minimizing expected squared alignment error given local history τti\tau_t^i is

C∗(τti)=E[gψ(o^ti)∣τti].C^*(\tau_t^i)=\mathbb{E}\left[g_\psi(\hat{o}_t^i)\mid\tau_t^i\right].

Alignment rewards the actor for retaining the projected team context it can infer locally. Policy guidance and the return objective train the same recurrent parameters for cooperative decisions (Eq. 11).

AuthorsOpenReview ↗

Response to Reviewer SDjH (Part 2/3)

W2: In the same actor, does supervising the updated memory (CtC_t) give a different result from supervising the decision feature (ztz_t)?

We held the single-token actor fixed at K=1K=1, d=24d=24, with the same projector architecture, 24-dimensional target and alignment weight. With supervision on ztz_t, the curve finishes above its CtC_t counterpart on all four tasks. The policy head reads ztz_t directly, so this loss acts on the feature used for the current action. Its gradients also reach the shared retention parameters that update memory for later steps. Fig. R1, p. 8

The decision feature ztz_t stays dd-dimensional as KK grows. The memory CtC_t has KK persistent dd-dimensional slots, each of which receives a projected target and feeds into the next step. Holding the supervision site at CtC_t and increasing KK from one to four improves late-training performance on all four tasks. Full DSM leads on CoordSum 5x20-80 and SMAX 5m_vs_6m. On RWARE medium-4ag-hard and large-4ag-hard, the single-token ztz_t variant has the higher final plotted point estimate, and its final 95% interval overlaps full DSM’s on each task. Thus Fig. R1 supports direct supervision of the readout at K=1K=1 and shows what four supervised memory slots add over one. The comparison between supervision sites was run at K=1K=1 only.

Q2: Do the multi-token gains persist when recurrent-state size and key–value write magnitude are both controlled?

On CoordSum 3x30-50, the four-token actor at d=16d=16 finishes ahead of the single-token actor at d=24d=24 while storing 320 versus 600 recurrent scalars, about 47% fewer. At fixed d=24d=24, four tokens store 672 scalars, only 12% more than one token. Four tokens also give a stronger late-training aggregate curve than eight or sixteen, both of which have more storage (Fig. 6a). These comparisons show that allocating more scalars alone cannot explain the pattern.

For one head and layer, the count is Kd+d2Kd+d^2 per agent, covering the tokens and retention matrix. Reducing dd narrows both. In the per-task runs underlying Fig. 6(a,c), the 320-scalar actor is level with the 600-scalar actor on LBF 15x15-4p-3f and trails it on SMAX 3s5z_vs_3s6z and RWARE large-8ag-hard. The gain with less storage is specific to CoordSum in this comparison. The separate width sweep shows a more modest aggregate benefit from increasing dd (Fig. 6c).

Eq. (7) distinguishes write magnitude from stored-state size. Before clipping, the new write is Δht=Kt⊤Vt=∑j=1K+1kt,jvt,j⊤\Delta h_t=K_t^\top V_t=\sum_{j=1}^{K+1}k_{t,j}v_{t,j}^{\top}, writing each key and value as a column vector. Its rank is at most min⁡(d,K+1)\min(d,K+1). More positions allow more write directions, while the learned keys and values determine the norm. RMS clipping bounds the stored state after adding the write to the decayed state. The sweeps did not independently match per-step write norms or hold them fixed together with total storage.

AuthorsOpenReview ↗

Response to Reviewer SDjH (Part 3/3)

Q3: How do runtime memory, inference latency and training cost compare with MAGPO?

Table 10 (Appendix F, p. 26) covers 44 tasks. On an RTX 3090, one actor call processes a full team in one environment. Buffer sizes (excluding aliases) and forward times are geometric means.

MetricMAGPO (H=128)DSM
Deployed actor parameters133.0K41.3K
Compiled actor buffers640.9 KiB295.7 KiB
Forward time per actor call94.5 μs114.7 μs
Median peak training VRAM above idle1,218 MiB1,722 MiB

Only the actor is deployed. DSM has 69% fewer actor parameters and uses smaller buffers, while each forward call takes 1.21 times as long. The median within-task DSM/MAGPO peak training VRAM ratio is 1.23. Training takes about 20–40% longer by task, which we attribute mainly to feature projection and distillation. On RWARE, DSM learns faster (Fig. 3) and reaches a final-policy IQM of 0.76 versus 0.64 (Table 1).

Q4: What representations do GRU+CTD and QMIX distill, and which hyperparameters did you use for each integration task?

  • GRU+CTD (Sec. 4.3). A two-layer projector (hidden width 128) maps the detached teacher encoder feature to gψ(stopgrad⁡(o^ti))∈R64g_\psi(\operatorname{stopgrad}(\hat{o}_t^i))\in\mathbb{R}^{64}. This target supervises the raw output of the H=64H=64 GRU before the policy torso.
  • DSM–MAPPO (Sec. 4.5). The projected centralized critic feature supervises the updated tokens. The deployed policy head reads the local decision feature ztiz_t^i.
  • DSM–QMIX (Sec. 4.5). Reshape the mixer’s first-layer weights W1(st)=∣hyper⁡w1(st)∣W_1(s_t)=\lvert\operatorname{hyper}_{w1}(s_t)\rvert to one row per agent. The projector maps the detached row W1(st)[i,:]W_1(s_t)[i,:] to targets for agent ii’s updated tokens. The joint-Q Bellman loss trains the mixer (Appendix B.5), and CTD updates the projector and local utility network. At execution, each agent acts greedily from its local Q-values.

On four SMAX tasks both integrations reach high win rates earlier. The original learners eventually approach similar performance. Fig. 8, p. 9

These per-task integration settings follow Sable’s frozen release. Appendix E.3 gives the published search ranges. Sable hyperparameter release

SMAX taskMAPPO actor / critic LRPPO epochs / minibatchesPPO clip / entropy coeff.Max. grad. norm / chunk lengthQMIX LRQMIX batch / replay size / target-update period
2s3z2.5e-4 / 1e-48 / 80.2 / 0.015 / 323e-432 / 2000 / 400
3s5z2.5e-4 / 2.5e-48 / 40.1 / 00.5 / 83e-5128 / 2000 / 400
3s_vs_5z2.5e-4 / 5e-48 / 40.05 / 1e-50.5 / 643e-564 / 2000 / 200
6h_vs_8z2.5e-4 / 2.5e-48 / 20.05 / 010 / 323e-4128 / 4000 / 100

Both learners use discount 0.99. MAPPO uses GAE λ=0.95\lambda=0.95, rollout length 128 and value-loss weight 0.5. Its baseline GRU width is 128. QMIX uses sequence length 5, exploration floor 0.05, gradient-norm limit 10 and mixer width 32. DSM defaults (Table 5) are K=4K=4, d=24d=24, one retention layer and head, RMS limit 50, and width 128 for the observation encoder, policy torso and projector hidden layer. CTD uses alignment weight 0.10, variance weight 0.05 and standard-deviation target 0.20.

Thank you for suggesting this comparison. Direct supervision of the decision feature works well with one slot. Supervising four persistent slots improves all four tasks over single-slot memory supervision and gives the highest final scores on CoordSum and SMAX (Fig. R1). Those slots also become distinguishable during training (Fig. R2). We hope this evidence addresses your concerns and that you will reconsider your rating.

Reviewer_x6weOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Decentralized Sequence Modeling (DSM) for cooperative multi-agent reinforcement learning. Its actor maintains persistent coordination tokens and a recurrent retention state, which are updated using local observations and previous actions to preserve information relevant to inter-agent coordination. During centralized training, coordination-token distillation supervises the updated tokens using representations extracted by a centralized teacher from joint observations. The teacher also provides action-distribution guidance, while execution relies solely on each agent's local history. Experiments across 44 tasks in six benchmarks demonstrate strong overall performance, with substantially fewer deployed actor parameters than the MAGPO's GRU actor and faster learning in terms of environment interactions on several tasks.

Critical strengths

1.The proposed actor memory design is elegant. It combines persistent coordination tokens updated through a retention layer with centralized representation supervision during training. This allows each actor to learn coordination-relevant memory while relying only on its local history during execution. I find this combination sufficiently novel and practically meaningful.

  1. The experimental results show strong performance against the evaluated baselines, substantially fewer deployed actor parameters than MAGPO's GRU actor, and faster learning in terms of environment interactions on several tasks. These results demonstrate the effectiveness of the proposed design and are an important reason for my positive assessment.
Critical weaknesses
  1. Although Figure 5 includes DSM w/o CTD, I think this comparison would be useful to include in Table 1. This would allow readers to see more directly the performance gains from the CTA memory architecture and the further improvements brought by distillation across the benchmark suites. The current ablations show substantial benefits on some tasks, but it remains unclear how consistently the two components contribute across different environments.

  2. The discussion of the main experimental results would benefit from more analysis of performance differences across environments, beyond describing the results already shown in the figures and tables. For example, why does Sable perform better on MPE, while DSM performs better in some environments where Sable is weaker? Further analysis would help readers understand which task characteristics favor DSM and where its limitations lie.

Questions for authors
  1. Does DSM w/o CTD consistently improve upon MAGPO across the benchmark suites in Table 1, and how much additional improvement does CTD provide? If results beyond the four tasks in Figure 5 are available, could the authors report them to clarify the respective contributions of the CTA architecture and CTD?

  2. What task characteristics might explain why Sable performs better on MPE, while DSM performs better in some other environments? Could the authors provide analysis supporting their interpretation of these differences?

2 comments
AuthorsOpenReview ↗

Response to Reviewer x6we (Part 1/2)

Thank you for asking how much the actor and distillation each contribute, and why the comparison with Sable varies across tasks. We extended the two ablations below to all 44 tasks and revisited the MPE and Connector comparisons.

To make Section 4 easier to navigate, we added a fourth research question about what the tokens learn and where to apply supervision. Section 4.2 reports performance and learning speed. Section 4.3 tests the memory architecture and supervision location, including a four-task comparison of supervising the updated memory or the current decision feature in a single-token actor (Fig. R1, p. 8). Section 4.4 tests how shuffling teacher targets across agents or timesteps affects learning on four tasks (Fig. 7) and examines how the token slots separate during training and how held-out alignment error relates to return (Fig. R2, p. 9).

W1. The four tasks in Fig. 5 make it difficult to see how consistently the actor architecture and distillation contribute across suites. Can the main comparison include DSM w/o CTD?

To keep Table 1 readable within the ten-page main paper, we kept its seven methods (DSM and six baselines) and placed the expanded comparison in Table 3 (Appendix A.4, p. 18). Table 3 reports all seven alongside DSM w/o CTD and DSM (1 token) on the same six suites. DSM w/o CTD keeps the multi-token actor, MAGPO's policy guidance, and the PPO auxiliary objective. Comparing it with MAGPO tests the change from a GRU to token memory under the same form of policy guidance. Comparing it with full DSM tests what coordination-token distillation (CTD) adds to that actor. The single-token variant retains CTD and tests the contribution of maintaining multiple tokens. The original four-task comparison also includes a GRU trained with CTD. Fig. 5, p. 7

The entries below reproduce the relevant columns of Table 3. Each entry is the IQM ± its bootstrap standard deviation. The first five suites use normalized returns, and SMAX uses win rates.

SuiteMAGPODSM w/o CTDDSM (1 token)DSM
CoordSum0.89 ± 0.0040.91 ± 0.0090.95 ± 0.0060.97 ± 0.003
LBF1.00 ± 0.0001.00 ± 0.0001.00 ± 0.0001.00 ± 0.000
Connector0.94 ± 0.0010.94 ± 0.0020.94 ± 0.0030.95 ± 0.001
MPE0.84 ± 0.0040.85 ± 0.0040.85 ± 0.0030.86 ± 0.002
RWARE0.64 ± 0.0060.68 ± 0.0100.72 ± 0.0060.76 ± 0.005
SMAX0.95 ± 0.0030.96 ± 0.0060.97 ± 0.0040.98 ± 0.001

Q1. Does the actor without CTD improve on MAGPO across suites, and how much does adding CTD help?

At the two decimals shown in Table 3, DSM w/o CTD improves on MAGPO in four suites and ties it in two. Its IQM gains are +0.04 on RWARE, +0.02 on CoordSum, and +0.01 each on MPE and SMAX. The two methods tie on LBF (1.00) and Connector (0.94). Adding CTD to the same token actor raises the IQM by 0.08 on RWARE, 0.06 on CoordSum, 0.02 on SMAX, and 0.01 each on MPE and Connector. LBF remains at 1.00. On RWARE, retaining CTD but reducing the memory to one token gives 0.72, between the no-CTD actor (0.68) and full DSM (0.76).

The learning curves show when the gains emerge. Full DSM reaches high returns earlier on CoordSum and RWARE and high win rates earlier on SMAX. Its RWARE lead persists through training. On Connector the curves stay close and their ordering changes during training. Fig. 13, p. 18

AuthorsOpenReview ↗

Response to Reviewer x6we (Part 2/2)

W2. The suite rankings show where the methods differ, but the paper gives little analysis of the task-level results.

We revisited MPE and Connector to check the effect of observation settings. The task scores below are the final policies' mean native episode returns over ten seeds after 20 million environment steps (Table 2, p. 15).

Connector. We discovered that DSM had used the old observation settings on con-15x15x23a. We reran DSM on that task with the updated Mava observations, which give each agent its own position and destination and the nearby paths and targets. The correction raises DSM's final-policy return from the originally reported 0.40 to 0.65, ahead of MAGPO (0.63) and below Sable (0.70). All four Sable scores on Connector retain their published values. DSM leads Sable on the 3-agent and 5-agent maps (0.88 vs. 0.85 and 0.81 vs. 0.79), and both score 0.74 on the 10-agent map. Sable leads on the 23-agent map (Table 1, note 2, and Table 2).

MPE. We reran recurrent Sable with the same Mava version and MPE observation settings used for DSM (Table 1, note 1). The Sable MPE suite IQM changes from 0.94 in the original Table 1 to 0.86 in the revised Table 1. Revised Tables 1–3 use the reruns for MPE. Sable leads DSM on three-agent simple_spread (−5.00 vs. −5.66) and five-agent simple_spread (−18.99 vs. −19.24). On ten-agent simple_spread, DSM leads (−39.05 vs. −39.99). Official Sable release

DSM achieves the highest CTDE aggregate on all six suites and remains best or tied for best when the centrally executed baselines are included. It leads outright on CoordSum, RWARE, and SMAX. At Table 1's displayed precision, it ties for best on LBF (1.00), Connector (0.95), and MPE (0.86). Fig. 3, p. 6 and Fig. 11, p. 16 show the suite and task learning curves.

Q2. What features of these tasks might favor Sable or DSM, and what evidence supports that interpretation?

Coordinating moves within a timestep. Two agents approaching the same landmark in simple_spread can collide and leave another landmark uncovered. In Connector, occupying a narrow passage can cut off another agent's route (Appendices C.3–C.4). Sable sees joint observations and can condition each action on teammates' preceding actions in the same timestep. That could help it resolve competing moves on three-agent spread and the 23-agent Connector map, where it leads. The reversal on ten-agent spread and the smaller Connector maps shows why team size alone gives an incomplete explanation.

Coordinating choices across time. In CoordSum, agents must reach a public target sum. They earn more when the history-based guesser also fails to predict the first agent's choice. Varying that choice changes what the other agents must contribute. In RWARE, a robot may need to yield in a shared aisle and then resume its shelf delivery (Appendices C.1 and C.5). Both methods have temporal memory. DSM trains each agent's persistent tokens to match the centralized teacher's representation for that agent at that timestep. The agent carries the updated tokens into its next local decision. DSM leads Sable on CoordSum (0.97 vs. 0.51) and RWARE (0.76 vs. 0.68). Removing CTD yields the two largest drops on these same suites (Q1). Figure 7 tests the role of matching directly. Shuffling targets across agents or timesteps weakens performance on each of the four tasks tested, including both RWARE layouts. Together these findings support a contribution from supervision matched to the agent making the decision and the timestep when it acts.

Thank you again for your constructive feedback. If the 44-task ablations (Table 3), the task-level analysis, and the clearer structure of Section 4 address your concerns, we would appreciate your considering higher presentation and overall scores.

Reviewer_8L4qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DSM, a CTDE method that replaces the GRU in each decentralized actor with a retention-based actor holding K persistent latent “coordination tokens.” These tokens are trained by regressing them onto projected features of a centralized sequence-model teacher, on top of MAGPO’s policy guidance. It is evaluated on 44 tasks across six benchmarks against MAGPO, CTDS, HAPPO, MAPPO, MAT, and Sable, with ablations on memory structure and target matching, plus a MAPPO/QMIX integration. The authors report the best CTDE aggregate on all six benchmarks and about 68% fewer deployed actor parameters than MAGPO’s GRU actor.

Critical strengths

The evaluation is broad and transparent: 44 tasks, ten seeds, rliable-style aggregates, and full per-task tables and curves in the appendix. This matters because the weaknesses below can be checked against the authors’ own data.

The target-shuffling experiment (time- and agent-shuffled CTD) is a useful control. It suggests the gain depends on matched teacher targets rather than on having any auxiliary loss. Execution also stays fully local.

Critical weaknesses

The gains are not cleanly attributed to the proposed mechanism. DSM bundles several changes relative to MAGPO: a GRU-to-retention swap, K latent tokens, feature distillation, and a VICReg-style variance floor. The claim that tokens “represent cooperative behavior” is never tested, with no probing or analysis of what they encode. The distillation target is a function of other agents’ observations that the actor cannot see, so the regression can at best recover a conditional expectation, which the authors acknowledge only in future work. The ablations cover four tasks with five seeds and several overlapping intervals. They are also confounded: The single-token variant has less memory unless d was scaled.

The shuffle control omits the “w/o CTD” and MAGPO references. On large-4ag-hard the shuffled variants are near zero, below MAGPO’s 4.56, so wrong targets look actively harmful rather than merely unhelpful. The closest prior work, PTDE, and the representation-learning methods in the related work (MA2E, MA2CL, TIMAR) are cited but not compared.

The headline claims are also stronger than the tables. “Best or tied-best on five benchmarks” includes LBF (saturated at 1.00 for four methods) and Connector (tied at 0.95). DSM loses to Sable on all three MPE tasks. On con-15x15x23a it scores 0.40 against 0.63 for MAGPO and 0.70 for Sable, which the text describes only as “Sable remains stronger.” That is a large drop below DSM’s own policy-guidance baseline, and it is not discussed. Suite IQMs with tiny intervals hide such task-level losses. The “68% fewer parameters” is measured against a GRU actor whose width is not stated and for which Table 7 lists no size search. Training cost (teacher plus projector plus extra loss) and inference latency of the sequential retention actor are not reported.

Questions for authors

Q1. Teacher features depend on joint observations the actor never sees. What do the tokens actually encode? Does lower L_align on held-out data correlate with return?

Q2. Does the single-token ablation use d=24 or d=96? Is GRU+CTD matched in hidden size and parameters?

Q3. Can Fig. 7 include “DSM w/o CTD” and MAGPO? Are shuffled targets worse than no CTD?

Q4. Why does DSM fall to 0.40 on con-15x15x23a, below MAGPO’s 0.63? Is this a scaling issue with team size or the RMS clamp?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8L4q (Part 1/4)

Thank you for the careful review. Your question uncovered an observation mismatch on 23-agent Connector. We corrected the run, extended the ablations to 44 tasks, and measured training and deployment costs.

Q4. Why did DSM score 0.40 on con-15x15x23a, below MAGPO's 0.63? Could team size or RMS clamping explain the drop?

Our original DSM run gave each agent the legacy 15×15×5 grid, flattened to 1,125 inputs. We reran DSM with the updated Mava observation. Each agent receives four coordinates for the agent and its goal, 25 obstacle features and 25 target features in a local 5×5 window, and a 23-dimensional agent ID (77 inputs in total). The team size, 128-wide observation encoder, and four 24-dimensional tokens stayed fixed. The corrected final-policy mean return is 0.65 ± 0.01 for DSM, compared with 0.63 ± 0.02 for MAGPO and 0.70 ± 0.01 for Sable (Table 2, p. 15, mean ± SD over ten seeds). Table 1, note 2 (p. 6) documents the correction. Fig. 11, p. 16

Table 2 gives native returns for larger SMAX teams. On 27m_vs_30m, DSM, MAGPO, and Sable score 1.99, 1.99, and 2.00, respectively. On smacv2_20_units they score 1.64, 1.56, and 1.47. In the separate four-task sensitivity study, RMS limits of 50 and 200 give similar attained performance. That sweep did not include the 23-agent Connector task. Fig. 6(d), p. 8

W3.1. Do the headline rankings reflect ties and task-level differences on MPE and Connector?

We reran Sable with its released per-task hyperparameters in DSM's version of Mava. This gives DSM and Sable the same updated MPE observations. Table 1, note 1 (p. 6) identifies the reruns, and Appendix C.7 documents the fixed normalization bounds. Table 2 shows how the ranking changes with team size. On five-agent spread Sable scores −18.99 to DSM's −19.24, while on ten-agent spread DSM scores −39.05 to Sable's −39.99. Sable also leads on the largest Connector task. Sable's frozen code

With these MPE and Connector corrections, DSM has the highest CTDE aggregate on all six suites and is best or tied for best on all six when CTCE methods are included (Table 1, p. 6). At the displayed precision, DSM and Sable tie on MPE (0.86) and Connector (0.95), and several methods tie on LBF (1.00). DSM's largest gains over MAGPO are on CoordSum (0.97 vs. 0.89) and RWARE (0.76 vs. 0.64). Among the five methods in Fig. 1, DSM ranks first on 32 of 44 tasks, with 29 outright wins and three exact ties before rounding (Appendix C.7). Fig. 1, p. 2 · Fig. 3, p. 6

W3.2. Why was GRU width absent from the search space? Which width and what training and inference costs did you compare?

We kept the hidden width fixed at 128, as in MAGPO's released configuration. Table 8 in the revision (formerly Table 7) lists the baseline hyperparameter search, which did not vary GRU width. Even against GRU64+CTD, DSM uses 27.4% fewer deployed weights on 5m_vs_6m (Q2 below). Table 10 (Appendix F, p. 26) covers 44 tasks on RTX 3090 GPUs, with one team per actor call.

MetricMAGPO (H = 128)DSM
Deployed parameters133.0K41.3K
Compiled buffers640.9 KiB295.7 KiB
Forward time per actor call94.5 µs114.7 µs
Peak training VRAM above idle1,218 MiB1,722 MiB

DSM uses about 69% fewer deployed parameters and 54% less buffer storage. Its forward call takes 20.2 µs longer. Buffers and timings are geometric means. Peak training memory is the median after warm-up, with a median within-task DSM/MAGPO ratio of 1.23. Training takes approximately 20–40% longer, depending on the task. We attribute much of the overhead to feature projection and distillation. The teacher and projector are removed at deployment.

AuthorsOpenReview ↗

Response to Reviewer 8L4q (Part 2/4)

W1. DSM changes the actor architecture, token count, and supervision objective. Given the limited task coverage and overlapping intervals, what can be attributed to each component? How much does the variance floor contribute?

We extended the single-token and no-CTD ablations to all 44 tasks. Table 3 (Appendix A.4, p. 18) gives the final-policy suite IQMs and their bootstrap standard deviations. The first five suites use normalized episode return and SMAX uses win rate.

SuiteDSMDSM, one tokenDSM w/o CTDMAGPO
CoordSum0.97 ± 0.0030.95 ± 0.0060.91 ± 0.0090.89 ± 0.004
LBF1.00 ± 0.0001.00 ± 0.0001.00 ± 0.0001.00 ± 0.000
Connector0.95 ± 0.0010.94 ± 0.0030.94 ± 0.0020.94 ± 0.001
MPE0.86 ± 0.0020.85 ± 0.0030.85 ± 0.0040.84 ± 0.004
RWARE0.76 ± 0.0050.72 ± 0.0060.68 ± 0.0100.64 ± 0.006
SMAX0.98 ± 0.0010.97 ± 0.0040.96 ± 0.0060.95 ± 0.003

Actor architecture. DSM w/o CTD keeps the four-token actor, MAGPO's policy guidance, and the PPO auxiliary objective. Comparing it with MAGPO shows that the change from a GRU to the token actor improves the final score on CoordSum and RWARE even without representation supervision. Their scores tie at the displayed precision on LBF and Connector.

Token count and supervision. Comparing full DSM with the single-token actor holds token width and the form of CTD fixed. Four tokens give higher final scores on five suites, with a displayed tie on LBF. Comparing full DSM with DSM w/o CTD holds the actor fixed and measures the effect of adding CTD. The clearest gains are on CoordSum and RWARE. Fig. 13 also shows that DSM reaches high returns earlier on those two suites and high win rates earlier on SMAX. The four-task GRU+CTD comparison in Fig. 5 gives a complementary check. Multiple tokens help on 5m_vs_6m, while GRU+CTD finishes ahead on 3s5z_vs_3s6z. Some individual curve bands overlap. The suite-wide ablations show where gains recur, and the fixed-actor intervention in Fig. 7 tests target correspondence directly. Fig. 5, p. 7 · Fig. 13, p. 18

Variance floor. The actor and projector learn together, so both can minimize the alignment error in Eq. (8) by producing constant outputs. Eq. (9) discourages this collapse by penalizing coordinates on either side whose standard deviation across samples falls below the target. The no-CTD ablation measures the full objective's contribution. Removing just the variance penalty while retaining alignment would isolate its individual effect, which we have not measured. In Fig. 7 (p. 8), both shuffles retain the floor and the alignment loss. Each falls below the no-CTD actor on all four tasks. Because the matched and shuffled runs use the same regularizer, the drop identifies a contribution from pairing the right agent and timestep.

AuthorsOpenReview ↗

Response to Reviewer 8L4q (Part 3/4)

Q1. What coordination information can a local actor learn from joint observations? And does better held-out alignment track higher return?

The teacher encodes joint observations into one feature per agent before decoding actions. CTD projects that feature onto the corresponding agent's updated tokens Cti,1:KC_t^{i,1:K}. The policy head reads ztiz_t^i for the current action. The tokens and retention matrix htih_t^i carry information into the next decision. At execution their updates depend only on the agent's local history.

For a fixed teacher and projector, the squared-error predictor that minimizes expected alignment loss given local history is the conditional mean E[gψ(o^ti)∣τti]\mathbb{E}[g_\psi(\hat{o}_t^i)\mid\tau_t^i]. Variation in the projected target that the local history cannot predict contributes irreducible error. Policy guidance and the return objective in Eq. (11) shape the same memory for action selection.

Fig. R2 examines two properties of this memory. On SMAX 6h_vs_8z, the four slots overlap in the initial t-SNE projection and occupy distinct regions after training (a,b). Panels (c–f) pair held-out alignment error with decentralized return. Within each task, each point represents an independent training run evaluated at the same training step. The Pearson correlations are −0.995 on CoordSum 8x15, −0.942 on RWARE large-8ag-hard, −0.922 on RWARE large-4ag-hard, and −0.965 on SMAX 10m_vs_11m. Closer alignment accompanies higher return in all four panels. The slots thus develop distinguishable representations, and the accuracy of the learned target predictions tracks cooperative performance. Fig. R2, p. 9

Q2. How closely are the memory ablations matched for capacity? What width does the single-token actor use, and how does GRU+CTD compare with DSM?

The single-token actor uses K=1K=1, d=24d=24. Full DSM uses K=4K=4, d=24d=24. With one head and one layer, the persistent state holds KdKd token entries and d2d^2 retention-matrix entries per agent.

Actor settingToken entriesMatrix entriesTotal recurrent entries
Full DSM, K = 4, d = 2496576672
Single-token ablation, K = 1, d = 2424576600
Hypothetical token-storage match, K = 1, d = 96969,2169,312

Holding width fixed preserves the retention dimensions and reduces total recurrent storage by 10.7%. Increasing the single token to width 96 would match the token-vector storage while making the retention matrix sixteen times larger.

GRU+CTD uses H = 64. On 5m_vs_6m, the deployed actors have 47,587 parameters for DSM, 65,547 for GRU64+CTD, and 135,435 for MAGPO's released GRU128. With the same encoder and policy head, a width-32 GRU would have 39,819 parameters. DSM thus uses 27.4% fewer weights than GRU64+CTD. During training, the projector maps the detached teacher feature to a target for the GRU hidden output before the policy torso.

Fig. R1 (p. 8) holds the single-token actor fixed and varies the supervision site. Supervising ztz_t gives higher final scores than supervising CtC_t on all four tasks. The policy head reads ztz_t directly, so its alignment loss acts on the current decision feature and also sends gradients through the shared retention layer. Increasing KK leaves the decision feature ztz_t at width dd and expands the persistent memory CtC_t to KK vectors of that width. Under CtC_t supervision, going from one token to four improves all four outcomes. Full DSM leads on the CoordSum and SMAX tasks. The single-token ztz_t variant leads on both RWARE tasks, where its uncertainty bands overlap DSM's.

AuthorsOpenReview ↗

Response to Reviewer 8L4q (Part 4/4)

W2.1. How should we interpret the near-zero returns under target shuffling on large-4ag-hard?

The comparison with DSM w/o CTD in revised Fig. 7 (p. 8) resolves the ambiguity in the original figure. The no-CTD actor learns to earn positive returns on this task. Both shuffled variants stay near zero, while correctly matched CTD brings the full actor above both no CTD and MAGPO. Pairing a local memory update with the wrong teacher context can therefore be more harmful than omitting representation supervision altogether.

Q3. Can Fig. 7 include DSM w/o CTD and MAGPO? Are the shuffled targets worse than no CTD?

Yes. Fig. 7 now shows full DSM, DSM w/o CTD, MAGPO, Sable, time-shuffled CTD, and agent-shuffled CTD. The time shuffle permutes targets across timesteps within an episode fragment. The agent shuffle permutes them across agents at a fixed timestep. Both keep the sampled targets, actor architecture, and matching objective, while breaking a different part of the target assignment.

Both shuffles finish below DSM w/o CTD on all four tasks. Their ordering relative to MAGPO depends on the task. Both finish below MAGPO on RWARE large-4ag-hard and SMAX 10m_vs_11m, and above MAGPO on CoordSum 3x30 and RWARE small-4ag-hard. The no-CTD actor itself finishes above MAGPO on three of the four tasks. These controls explain why the no-CTD actor is the informative reference for the effect of misleading targets.

W2.2. Why were PTDE, MA2E, MA2CL, and TIMAR discussed in related work but omitted from the experimental comparison?

Our JAX/Mava evaluation includes two distillation baselines with distinct targets. MAGPO guides the local action distribution, giving us a common policy objective while we change the actor memory and add CTD. CTDS distills centralized action values into local utilities. PTDE is the closest comparison in feature space. It trains a local predictor of a personalized centralized feature used for decisions. DSM supervises the updated tokens carried into the next decision. Table 4 (Appendix B.7, p. 21) sets out the supervision targets and execution inputs of PTDE, DSM, MAGPO, and CTDS.

MA2E reconstructs missing teammate information, MA2CL uses masked contrastive learning, and TIMAR predicts joint transitions (§5, p. 10). We prioritized controlled comparisons of the actor, supervision target, and policy guidance in the common training setup. The revision compares the four methods you named at the level of their learning objectives. A direct performance comparison, particularly with PTDE, would require implementing or adapting and validating their training objectives in that setup. The empirical ranking in Tables 1–2 covers the methods we evaluated.

Thank you for pressing us on soundness. We corrected the Connector observation and matched the DSM and Sable MPE evaluations (Tables 1–2, pp. 6, 15). The 44-task ablations (Table 3, p. 18), the shuffle controls and supervision-site comparison (Figs. 7 and R1, p. 8), the token analysis (Fig. R2, p. 9), and the training and inference costs (Table 10, p. 26) address the individual questions you raised. We hope you will reconsider your assessment in light of these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.