- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Decentralized Sequence Modeling (DSM), combining a retention-based Coordination-Token Actor (CTA) with Coordination-Token Distillation (CTD). Building on MAGPO’s policy-guidance framework, DSM supervises persistent local memory using projected centralized teacher features, while retaining fully decentralized execution. The empirical results are promising.
Critical strengths
- A meaningful extension from action guidance to recurrent-memory supervision.
- Broad empirical evaluation with substantial improvements on challenging tasks.
Critical weaknesses
- It is unclear whether CTD preserves useful coordination information rather than merely making the representations easier to match.
In Eq. (8), both the actor tokens and the teacher-side projector are optimized. The alignment loss can therefore decrease either because the actor learns useful teacher information or because the projector discards information that is difficult to match. The variance penalty prevents constant outputs but does not establish that the remaining variation is relevant to coordination. This does not imply that the method degenerates in practice; rather, the paper needs more direct evidence about what information the tokens retain and use for decisions.
- The specific benefit of supervising persistent memory is not sufficiently isolated.
The GRU+CTD and no-CTD ablations provide useful evidence for the combined design. However, they do not directly test whether supervising the updated memory tokens is more effective than applying comparable representation supervision elsewhere in the same actor. A comparison between supervising and the decision representation , with comparable projector capacity and loss scaling, would clarify whether the gains specifically support memory supervision or primarily reflect the benefits of an auxiliary representation objective.
Questions for authors
-
Why is the agent-specific encoder feature an appropriate target for future local memory?
-
Do the multi-token gains persist after controlling for recurrent-state size and key–value write magnitude?
-
What are the runtime memory, inference latency, and training cost relative to MAGPO?
-
Could you specify the GRU+CTD and QMIX distillation interfaces and the final per-task hyperparameters?
3 comments
Response to Reviewer SDjH (Part 2/3)
W2: In the same actor, does supervising the updated memory () give a different result from supervising the decision feature ()?
We held the single-token actor fixed at , , with the same projector architecture, 24-dimensional target and alignment weight. With supervision on , the curve finishes above its counterpart on all four tasks. The policy head reads directly, so this loss acts on the feature used for the current action. Its gradients also reach the shared retention parameters that update memory for later steps. Fig. R1, p. 8
The decision feature stays -dimensional as grows. The memory has persistent -dimensional slots, each of which receives a projected target and feeds into the next step. Holding the supervision site at and increasing from one to four improves late-training performance on all four tasks. Full DSM leads on CoordSum 5x20-80 and SMAX 5m_vs_6m. On RWARE medium-4ag-hard and large-4ag-hard, the single-token variant has the higher final plotted point estimate, and its final 95% interval overlaps full DSM’s on each task. Thus Fig. R1 supports direct supervision of the readout at and shows what four supervised memory slots add over one. The comparison between supervision sites was run at only.
Q2: Do the multi-token gains persist when recurrent-state size and key–value write magnitude are both controlled?
On CoordSum 3x30-50, the four-token actor at finishes ahead of the single-token actor at while storing 320 versus 600 recurrent scalars, about 47% fewer. At fixed , four tokens store 672 scalars, only 12% more than one token. Four tokens also give a stronger late-training aggregate curve than eight or sixteen, both of which have more storage (Fig. 6a). These comparisons show that allocating more scalars alone cannot explain the pattern.
For one head and layer, the count is per agent, covering the tokens and retention matrix. Reducing narrows both. In the per-task runs underlying Fig. 6(a,c), the 320-scalar actor is level with the 600-scalar actor on LBF 15x15-4p-3f and trails it on SMAX 3s5z_vs_3s6z and RWARE large-8ag-hard. The gain with less storage is specific to CoordSum in this comparison. The separate width sweep shows a more modest aggregate benefit from increasing (Fig. 6c).
Eq. (7) distinguishes write magnitude from stored-state size. Before clipping, the new write is , writing each key and value as a column vector. Its rank is at most . More positions allow more write directions, while the learned keys and values determine the norm. RMS clipping bounds the stored state after adding the write to the decayed state. The sweeps did not independently match per-step write norms or hold them fixed together with total storage.
Response to Reviewer SDjH (Part 3/3)
Q3: How do runtime memory, inference latency and training cost compare with MAGPO?
Table 10 (Appendix F, p. 26) covers 44 tasks. On an RTX 3090, one actor call processes a full team in one environment. Buffer sizes (excluding aliases) and forward times are geometric means.
| Metric | MAGPO (H=128) | DSM |
|---|---|---|
| Deployed actor parameters | 133.0K | 41.3K |
| Compiled actor buffers | 640.9 KiB | 295.7 KiB |
| Forward time per actor call | 94.5 μs | 114.7 μs |
| Median peak training VRAM above idle | 1,218 MiB | 1,722 MiB |
Only the actor is deployed. DSM has 69% fewer actor parameters and uses smaller buffers, while each forward call takes 1.21 times as long. The median within-task DSM/MAGPO peak training VRAM ratio is 1.23. Training takes about 20–40% longer by task, which we attribute mainly to feature projection and distillation. On RWARE, DSM learns faster (Fig. 3) and reaches a final-policy IQM of 0.76 versus 0.64 (Table 1).
Q4: What representations do GRU+CTD and QMIX distill, and which hyperparameters did you use for each integration task?
- GRU+CTD (Sec. 4.3). A two-layer projector (hidden width 128) maps the detached teacher encoder feature to . This target supervises the raw output of the GRU before the policy torso.
- DSM–MAPPO (Sec. 4.5). The projected centralized critic feature supervises the updated tokens. The deployed policy head reads the local decision feature .
- DSM–QMIX (Sec. 4.5). Reshape the mixer’s first-layer weights to one row per agent. The projector maps the detached row to targets for agent ’s updated tokens. The joint-Q Bellman loss trains the mixer (Appendix B.5), and CTD updates the projector and local utility network. At execution, each agent acts greedily from its local Q-values.
On four SMAX tasks both integrations reach high win rates earlier. The original learners eventually approach similar performance. Fig. 8, p. 9
These per-task integration settings follow Sable’s frozen release. Appendix E.3 gives the published search ranges. Sable hyperparameter release
| SMAX task | MAPPO actor / critic LR | PPO epochs / minibatches | PPO clip / entropy coeff. | Max. grad. norm / chunk length | QMIX LR | QMIX batch / replay size / target-update period |
|---|---|---|---|---|---|---|
| 2s3z | 2.5e-4 / 1e-4 | 8 / 8 | 0.2 / 0.01 | 5 / 32 | 3e-4 | 32 / 2000 / 400 |
| 3s5z | 2.5e-4 / 2.5e-4 | 8 / 4 | 0.1 / 0 | 0.5 / 8 | 3e-5 | 128 / 2000 / 400 |
| 3s_vs_5z | 2.5e-4 / 5e-4 | 8 / 4 | 0.05 / 1e-5 | 0.5 / 64 | 3e-5 | 64 / 2000 / 200 |
| 6h_vs_8z | 2.5e-4 / 2.5e-4 | 8 / 2 | 0.05 / 0 | 10 / 32 | 3e-4 | 128 / 4000 / 100 |
Both learners use discount 0.99. MAPPO uses GAE , rollout length 128 and value-loss weight 0.5. Its baseline GRU width is 128. QMIX uses sequence length 5, exploration floor 0.05, gradient-norm limit 10 and mixer width 32. DSM defaults (Table 5) are , , one retention layer and head, RMS limit 50, and width 128 for the observation encoder, policy torso and projector hidden layer. CTD uses alignment weight 0.10, variance weight 0.05 and standard-deviation target 0.20.
Thank you for suggesting this comparison. Direct supervision of the decision feature works well with one slot. Supervising four persistent slots improves all four tasks over single-slot memory supervision and gives the highest final scores on CoordSum and SMAX (Fig. R1). Those slots also become distinguishable during training (Fig. R2). We hope this evidence addresses your concerns and that you will reconsider your rating.
Response to Reviewer SDjH (Part 1/3)
Thank you for distinguishing action guidance from supervision of memory used in later decisions. We reran Sable with the updated MPE observations and corrected DSM’s observations on
con-15x15x23a(Table 1, footnotes 1–2, p. 6).A projection can discard teacher information. Eq. (8) detaches the teacher feature before projection, so alignment updates the actor and projector. Eq. (9) penalizes low variance on both sides. We tested whether the variation retained in the targets matters for control.
We held the architecture, sampled targets and full CTD objective fixed and shuffled targets across timesteps within an episode fragment or across agents at the same timestep. On all four tasks, matched CTD finishes above the no-CTD actor and both shuffles finish below it. On RWARE
large-4ag-hard, both shuffles stay near zero. The gain depends on assigning each update the context of its own agent and timestep. Fig. 7, p. 8On SMAX
6h_vs_8z, the four slots overlap in the initial t-SNE projection and occupy distinct regions after training. Slot identity is thus visible in the learned representation. In Fig. R2(c–f), each point is an independently trained run evaluated at the same training step as the other runs on that task. Smaller held-out alignment errors accompany higher decentralized returns across runs. The Pearson correlations are −0.995 on CoordSum8x15, −0.942 and −0.922 on RWARElarge-8ag-hardandlarge-4ag-hard, and −0.965 on SMAX10m_vs_11m. Together with the shuffle control, these observations support the usefulness of the projected context for local decisions. Fig. R2, p. 9Table 3 (Appendix A.4, p. 18) covers all 44 tasks. Entries are final-policy suite IQMs ± bootstrap SDs, using normalized returns for the first five suites and win-rate fractions for SMAX.
On CoordSum and RWARE, removing CTD still leaves the token actor ahead of MAGPO. Supervising the same memory raises the IQM from 0.91 to 0.97 and from 0.68 to 0.76, respectively.
The teacher encodes joint observations into an agent-specific feature o^ti before decoding any current actions. CTD projects that agent’s team context onto its updated tokens Cti,1:K. At the next step, the actor reads those tokens with its new local observation, previous action and retention state to form zt+1i (Secs. 3.1–3.2).
For a fixed teacher and projector, the predictor minimizing expected squared alignment error given local history τti is
C∗(τti)=E[gψ(o^ti)∣τti].
Alignment rewards the actor for retaining the projected team context it can infer locally. Policy guidance and the return objective train the same recurrent parameters for cooperative decisions (Eq. 11).