- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ART trains a centralized Koopman autoencoder on joint transitions. Its latent coordinates for the current state are cached and used as auxiliary regression targets for decentralized actors. During MAPPO updates, a linear head on the actor backbone predicts these targets, and no gradient flows into the teacher. The teacher and the head are dropped at execution. The experiments cover nine Multi-Agent MuJoCo settings (task, agent partition and observation type), with six seeds in the main comparisons. The biggest gains are on Ant. Ablations with a fixed projection, value targets and different model losses test how much the representation targets matter.
Critical strengths
S1. A clean design that keeps execution decentralized: Eqs. (11)-(13) and Figure 2 clearly show how separately learned global targets shape the actor's features without becoming inputs at execution. Only the actor backbone and the auxiliary head get gradients; the teacher and the stored targets stay fixed during the policy update. The paper does not assume that full-state information can be recovered exactly under partial observation. Pairing an online bilinear Koopman teacher with a throwaway actor head is coherent, and the claim of no extra model at deployment follows directly from the design.
S2. Large gains on Ant, with useful controls: Table 1 uses six seeds and the same MAPPO hyperparameters for all methods. Full-observation Ant 8×1 goes from 2,862 to 5,040, and partial-observation Ant 2×4 from 1,454 to 3,267. Table 2 goes beyond the plain baseline with fixed-projection, value-target and teacher-loss controls on matched seeds. The appendix openly shows settings where simpler teacher objectives do as well, and the main text admits that the smaller HalfCheetah gains are within seed noise.
Critical weaknesses
W1. The novelty claim misses close prior work: The introduction says the paper is unique in combining independently learned dynamical coordinates with actor supervision used only during training. But I found three close papers.
- [LDD (NeurIPS 2022)] already adds feature matching to a frozen teacher, trained separately on dynamics, during actor–critic learning.
- [PTLD (March 2026), arxiv 2603.04531v1)] trains PPO together with stop-gradient matching from local actor features to privileged critic features.
- In MARL, [IWoL (September 2025), arxiv 2509.25550v1] uses privileged auxiliary supervision with decoders and communication used only in training, and decentralized execution in its implicit variant.
These methods are not the same as ART. LDD's teacher is pretrained offline and fixed, PTLD's teacher is trained on values, and IWoL has no independently trained predictive teacher. None of them is an exact copy of ART's bilinear Koopman construction. Still, together they make the conceptual contribution much narrower than the paper suggests, since the paper only compares itself with PPG and policy-input methods. If "dynamical coordinates" means bilinear Koopman coordinates specifically, the paper should say so, as it already does at lines 107-110. The general transfer idea alone cannot carry the novelty claim.
W2. The experiments do not clearly show the value of the predictive targets themselves:
- No comparably trained generic nonlinear dynamics teacher, and no additive linear controlled model, is tested.
- Replacing the learned encoder with a fixed projection removes representation learning as well as dynamics training, so it does not isolate the dynamics part (§5.2).
- In Table 2, fixed targets already recover about 68% (full-observation Ant 4×2) and 77% (partial-observation Ant 2×4) of ART's gain over MAPPO. These are descriptive ratios, not causal splits or significance tests. ART still beats these fixed targets by 613 and 534 return.
- Reconstruction-only targets give mixed evidence. They are clearly worse on partial-observation Ant 2×4 (2,957 vs 3,723), but better on full-observation Ant 2×4 (4,847 vs ART's 4,613; Appendix A.5).
The results support auxiliary supervision, and a predictive benefit that depends on the setting. Given the close prior work, I am not convinced that they show enough extra value from ART's specific target construction. A combination can be a contribution without every part being essential, but here the key target choice is not tested enough.
Questions for authors
Q1. Compared with LDD, PTLD and IWoL, is the novelty the specific online bilinear Koopman target and the unchanged-actor MARL setup, rather than the general idea of training-only transfer? What insight or practical advantage sets this design apart from a matched predictive-teacher alternative?
Q2. Please explain the target normalization and what information each control can access, including the contact features in Appendix A.2, and share paired-seed differences if you have them. The good no-contact result on full-observation Ant 4×2 is relevant here. If possible, compare against a generic nonlinear or additive linear transition model with the same transfer setup, encoder size, information, horizon and training budget. This is a suggested new comparison; I am not saying the paper already has it. A short measurement of training time and memory would also help judge whether the extra training cost is worth it.
2 comments
Assessment after the new experiments
I have read the new experiments. The matched MLP and additive-linear baselines help address my question about the desired prediction. These results provide further evidence for using a bilinear form under incomplete observations, though the advantage is not consistent across all environments. I also appreciate the comparisons with HAPPO, MA2CL and PPG, as well as the reported wall-clock times.
With the expanded discussion, I am now satisfied with the related work section. I especially appreciate the clarification that this work extends prior work on transferring representations at training time only to an online bilinear Koopman form.
A few minor suggestions:
- Given the claim that the method outperforms all baselines using transitions, it would be helpful to report the score differences across all matched trials.
- I would suggest limiting the conclusion about dropping the contact variables to the fully-observed tasks.
- It would be clearer to report the full range of training-time increases (15-29%) in Table 12.
None of these require additional experiments. Overall, the new results make me more convinced, and I will keep my score of 3 (Weak accept).
We sincerely thank the reviewer for the careful and detailed review, and for the constructive suggestions. The pointers to related work and the proposed transition-model controls have directly improved the paper, and we have run the suggested comparisons.
W1 & Q1: Prior Work & Advantage of Koopman.. We thank the reviewer for these references and have added LDD, PTLD and IWoL to the related work in the revised pdf (which will be uploaded shortly). We also revised the contribution statement to name the bilinear Koopman targets explicitly, as suggested. We agree that the training-only transfer of privileged information is established, and we do not claim it as our contribution. ART studies whether a latent in which the joint dynamics of all agents are linear, and driven by the joint action through a bilinear term, is a better auxiliary target for decentralized actors than an unstructured predictive latent. Koopman modelling of dynamics has several known properties that makes it a more suitable choice for our approach. A Koopman latent evolves linearly, so its predictive content lies in coordinates that the actor's linear auxiliary head can match directly. The bilinear term makes the effect of each agent's action depend on the current state, so the target encodes how teammates' actions shape each agent's future, which is what a decentralized actor cannot observe. A matched predictive model with an unstructured transition can carry the same information, but in coordinates with no such structural guarantee. The linear transition also has practical advantages: the operators are initialized in closed form by EDMD, and the eigenvalues of K can expose the stability of the learned dynamics which can be further used to analyse the structure of the learned dynamics, such as which modes persist and on what time scales.
W2 & Q2:
(i) Value of the Koopman Target. As suggested, to isolate the dynamics part, we therefore ran two additional controls that keep everything else fixed and change only the transition: a nonlinear MLP transition g(zt,at) with matched parameter count, and an additive linear transition (Nk=0). Both use the same encoder, decoder, losses, input, horizon and training budget as ART. Under partial observation, the bilinear transition performs better than both on every paired seed. Under full observation the controls match ART. This is the same pattern as for the reconstruction-only targets. When every actor observes the full state, any state-derived target gives a useful training signal. When it does not, the predictive structure of the target matters, which is the setting ART is designed for. We also compare ART with MA2CL and PPG, and apply ART to HAPPO. MA2CL reaches a higher return than ART under partial observation, but it trains the actor through an additional transformer over all agents and takes about 24% more training time than ART. PPG, which regresses the actor onto a value target, falls below HAPPO, while ART improves HAPPO under both partial and full observation. The results are shown in Tables G1–G4 of our general response.
(ii) Normalization and information. The Koopman target is the encoder output followed by a layer normalization without learned scale or offset, so each target vector has zero mean and unit variance. The encoder input is normalized per dimension with statistics fixed at the first model update. The value target is the GAE return standardized across each rollout. Every target is computed from the same full training-time state, which the actor never receives. The Koopman encoder/decoder and both new controls also receive per-foot contact forces. The contact features do not drive the gain: removing them gives 5,335 ± 54 against 5,128 ± 223 on full-observation Ant 4×2. We have marked them as optional in our code and paper.
(iii) Training cost. On matched hardware , ART adds 15–17% training time over MAPPO (3.93 h vs 3.36 h on Ant 2×4, 7.01 h vs 6.12 h on Ant 4×2), with unchanged peak memory (9.5 GiB), and nothing at execution (Table G6 in our general response).