acceptodds
Under review as a conference paper at ICLR 2027

ART: Auxiliary Representation Transfer for Multi-Agent Policy Optimization

Abstract

Actor-critic methods in multi-agent reinforcement learning (MARL) typically rely solely on scalar rewards to shape actor representations. This narrow training signal leaves rich joint system dynamics unexploited, which severely limits performance and adaptability, especially under partial observability. We overcome this by introducing Auxiliary Representation Transfer (ART), a novel framework that aligns decentralized actor features with the predictive latent coordinates of a centralized Koopman autoencoder. Unlike prior methods that feed global models directly into the policy, ART regresses these coordinates through a linear auxiliary head during training to embed global predictive dynamics directly into the action selection process. Crucially, this transfer requires no execution-time overhead and leaves the actors' local observations and network architectures entirely unchanged. Experiments on Multi-Agent MuJoCo demonstrate that ART substantially improves the sample efficiency and final performance of Multi-Agent Proximal Policy Optimization (MAPPO). Notably, ART yields up to a 76% performance gain on Ant under full observation and more than doubles the final return on Ant under partial observation.

Primary area:
reinforcement learning
Keywords:
Multi Agent Reinforcement Learning, Koopman Operator

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fy1bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ART trains a centralized Koopman autoencoder on joint transitions. Its latent coordinates for the current state are cached and used as auxiliary regression targets for decentralized actors. During MAPPO updates, a linear head on the actor backbone predicts these targets, and no gradient flows into the teacher. The teacher and the head are dropped at execution. The experiments cover nine Multi-Agent MuJoCo settings (task, agent partition and observation type), with six seeds in the main comparisons. The biggest gains are on Ant. Ablations with a fixed projection, value targets and different model losses test how much the representation targets matter.

Critical strengths

S1. A clean design that keeps execution decentralized: Eqs. (11)-(13) and Figure 2 clearly show how separately learned global targets shape the actor's features without becoming inputs at execution. Only the actor backbone and the auxiliary head get gradients; the teacher and the stored targets stay fixed during the policy update. The paper does not assume that full-state information can be recovered exactly under partial observation. Pairing an online bilinear Koopman teacher with a throwaway actor head is coherent, and the claim of no extra model at deployment follows directly from the design.

S2. Large gains on Ant, with useful controls: Table 1 uses six seeds and the same MAPPO hyperparameters for all methods. Full-observation Ant 8×1 goes from 2,862 to 5,040, and partial-observation Ant 2×4 from 1,454 to 3,267. Table 2 goes beyond the plain baseline with fixed-projection, value-target and teacher-loss controls on matched seeds. The appendix openly shows settings where simpler teacher objectives do as well, and the main text admits that the smaller HalfCheetah gains are within seed noise.

Critical weaknesses

W1. The novelty claim misses close prior work: The introduction says the paper is unique in combining independently learned dynamical coordinates with actor supervision used only during training. But I found three close papers.

  • [LDD (NeurIPS 2022)] already adds feature matching to a frozen teacher, trained separately on dynamics, during actor–critic learning.
  • [PTLD (March 2026), arxiv 2603.04531v1)] trains PPO together with stop-gradient matching from local actor features to privileged critic features.
  • In MARL, [IWoL (September 2025), arxiv 2509.25550v1] uses privileged auxiliary supervision with decoders and communication used only in training, and decentralized execution in its implicit variant.

These methods are not the same as ART. LDD's teacher is pretrained offline and fixed, PTLD's teacher is trained on values, and IWoL has no independently trained predictive teacher. None of them is an exact copy of ART's bilinear Koopman construction. Still, together they make the conceptual contribution much narrower than the paper suggests, since the paper only compares itself with PPG and policy-input methods. If "dynamical coordinates" means bilinear Koopman coordinates specifically, the paper should say so, as it already does at lines 107-110. The general transfer idea alone cannot carry the novelty claim.

W2. The experiments do not clearly show the value of the predictive targets themselves:

  • No comparably trained generic nonlinear dynamics teacher, and no additive linear controlled model, is tested.
  • Replacing the learned encoder with a fixed projection removes representation learning as well as dynamics training, so it does not isolate the dynamics part (§5.2).
  • In Table 2, fixed targets already recover about 68% (full-observation Ant 4×2) and 77% (partial-observation Ant 2×4) of ART's gain over MAPPO. These are descriptive ratios, not causal splits or significance tests. ART still beats these fixed targets by 613 and 534 return.
  • Reconstruction-only targets give mixed evidence. They are clearly worse on partial-observation Ant 2×4 (2,957 vs 3,723), but better on full-observation Ant 2×4 (4,847 vs ART's 4,613; Appendix A.5).

The results support auxiliary supervision, and a predictive benefit that depends on the setting. Given the close prior work, I am not convinced that they show enough extra value from ART's specific target construction. A combination can be a contribution without every part being essential, but here the key target choice is not tested enough.

Questions for authors

Q1. Compared with LDD, PTLD and IWoL, is the novelty the specific online bilinear Koopman target and the unchanged-actor MARL setup, rather than the general idea of training-only transfer? What insight or practical advantage sets this design apart from a matched predictive-teacher alternative?

Q2. Please explain the target normalization and what information each control can access, including the contact features in Appendix A.2, and share paired-seed differences if you have them. The good no-contact result on full-observation Ant 4×2 is relevant here. If possible, compare against a generic nonlinear or additive linear transition model with the same transfer setup, encoder size, information, horizon and training budget. This is a suggested new comparison; I am not saying the paper already has it. A short measurement of training time and memory would also help judge whether the extra training cost is worth it.

2 comments

We sincerely thank the reviewer for the careful and detailed review, and for the constructive suggestions. The pointers to related work and the proposed transition-model controls have directly improved the paper, and we have run the suggested comparisons.

W1 & Q1: Prior Work & Advantage of Koopman.. We thank the reviewer for these references and have added LDD, PTLD and IWoL to the related work in the revised pdf (which will be uploaded shortly). We also revised the contribution statement to name the bilinear Koopman targets explicitly, as suggested. We agree that the training-only transfer of privileged information is established, and we do not claim it as our contribution. ART studies whether a latent in which the joint dynamics of all agents are linear, and driven by the joint action through a bilinear term, is a better auxiliary target for decentralized actors than an unstructured predictive latent. Koopman modelling of dynamics has several known properties that makes it a more suitable choice for our approach. A Koopman latent evolves linearly, so its predictive content lies in coordinates that the actor's linear auxiliary head can match directly. The bilinear term makes the effect of each agent's action depend on the current state, so the target encodes how teammates' actions shape each agent's future, which is what a decentralized actor cannot observe. A matched predictive model with an unstructured transition can carry the same information, but in coordinates with no such structural guarantee. The linear transition also has practical advantages: the operators are initialized in closed form by EDMD, and the eigenvalues of K can expose the stability of the learned dynamics which can be further used to analyse the structure of the learned dynamics, such as which modes persist and on what time scales.

W2 & Q2:

(i) Value of the Koopman Target. As suggested, to isolate the dynamics part, we therefore ran two additional controls that keep everything else fixed and change only the transition: a nonlinear MLP transition g(zt,at)g(z_t, a_t) with matched parameter count, and an additive linear transition (Nk=0N_k = 0). Both use the same encoder, decoder, losses, input, horizon and training budget as ART. Under partial observation, the bilinear transition performs better than both on every paired seed. Under full observation the controls match ART. This is the same pattern as for the reconstruction-only targets. When every actor observes the full state, any state-derived target gives a useful training signal. When it does not, the predictive structure of the target matters, which is the setting ART is designed for. We also compare ART with MA2CL and PPG, and apply ART to HAPPO. MA2CL reaches a higher return than ART under partial observation, but it trains the actor through an additional transformer over all agents and takes about 24% more training time than ART. PPG, which regresses the actor onto a value target, falls below HAPPO, while ART improves HAPPO under both partial and full observation. The results are shown in Tables G1–G4 of our general response.

(ii) Normalization and information. The Koopman target is the encoder output followed by a layer normalization without learned scale or offset, so each target vector has zero mean and unit variance. The encoder input is normalized per dimension with statistics fixed at the first model update. The value target is the GAE return standardized across each rollout. Every target is computed from the same full training-time state, which the actor never receives. The Koopman encoder/decoder and both new controls also receive per-foot contact forces. The contact features do not drive the gain: removing them gives 5,335 ± 54 against 5,128 ± 223 on full-observation Ant 4×2. We have marked them as optional in our code and paper.

(iii) Training cost. On matched hardware , ART adds 15–17% training time over MAPPO (3.93 h vs 3.36 h on Ant 2×4, 7.01 h vs 6.12 h on Ant 4×2), with unchanged peak memory (9.5 GiB), and nothing at execution (Table G6 in our general response).

Reviewer_fy1bOpenReview ↗

Assessment after the new experiments

I have read the new experiments. The matched MLP and additive-linear baselines help address my question about the desired prediction. These results provide further evidence for using a bilinear form under incomplete observations, though the advantage is not consistent across all environments. I also appreciate the comparisons with HAPPO, MA2CL and PPG, as well as the reported wall-clock times.

With the expanded discussion, I am now satisfied with the related work section. I especially appreciate the clarification that this work extends prior work on transferring representations at training time only to an online bilinear Koopman form.

A few minor suggestions:

  1. Given the claim that the method outperforms all baselines using transitions, it would be helpful to report the score differences across all matched trials.
  2. I would suggest limiting the conclusion about dropping the contact variables to the fully-observed tasks.
  3. It would be clearer to report the full range of training-time increases (15-29%) in Table 12.

None of these require additional experiments. Overall, the new results make me more convinced, and I will keep my score of 3 (Weak accept).

Reviewer_p8TkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a centralized deep Koopman autoencoder on joint trajectories to find coordinates where the dynamics are roughly linear, then adds a small linear head on each decentralized actor that regresses against those coordinates (with stop-gradient). The actor's features pick up global dynamics information during training without any execution-time change. Plugged into MAPPO on Multi-Agent MuJoCo, it improves sample efficiency and return, with the largest gains under partial observation, and ablations compare the Koopman target against simpler alternatives.

Critical strengths

The idea is clean: instead of asking the actor to regress return-based targets , it regresses against coordinates from a dynamics model trained independently of the policy. The separation means the auxiliary signal carries joint-transition information, not just value estimates, and the linear head is truly additive, no execution-time overhead.

Critical weaknesses

Baselines are thin: the paper positions itself against PPG, TIMAR, MAPO-LSO, MA2CL, and MA²E, but only compares against MAPPO. There is no HAPPO run, no off-policy result, and no comparison to the obvious upper bound—giving the actor the true global state or a learned global encoder as input.

Questions for authors

What is the gap between ART and the upper bound where each actor receives the true global state (or a frozen global encoder) as input?

2 comments

We thank the reviewer for the encouraging comments on our approach and address the remaining concerns below.

W1 & Q1: More Comparisons & Upper Bound. Following your suggestion, we compare ART with a centralized MAPPO policy that receives the full state and controls all joints (centralized training and centralized execution, i.e. Ant 1×8). In this additional rebuttal experiment, the centralized policy reaches 4,690 ± 350 on Ant (6 seeds), while ART reaches 5,128 ± 163 on Ant 4×2 and 5,040 ± 319 on Ant 8×1 under full observation, with decentralized execution (Table G5 in our general response). Additionally, we run ART with HAPPO and compare it with PPG and MA2CL. ART improves HAPPO in every setting except HalfCheetah 3×2, where the two are within seed variation (Table G4). PPG falls below HAPPO, while MA2CL reaches a higher return than ART under partial observation, at 24% more training time (Table G3).

Hello Reviewer p8Tk22,

Thank you for your valuable feedback. We hope our additional results and responses have adequately addressed your concerns. As tomorrow is the last day to engage with reviewers, we wanted to send a note to see if we have successfully addressed your concerns. We would greatly appreciate any engagement and thank you for your feedback so far.

Reviewer_mW9POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Auxiliary Representation Transfer (ART), a framework that improves multi-agent policy optimization by transferring representations learned from a centralized Koopman autoencoder to decentralized actors. The autoencoder learns predictive latent representations through reconstruction, multi-step dynamics prediction, and variance regularization. A linear auxiliary head is trained to predict these representations from actor features, shaping the actor backbone during training without changing its execution-time architecture. Experiments on nine Multi-Agent MuJoCo configurations show improvements over MAPPO in final returns and sample efficiency, particularly on Ant.

Critical strengths
  1. The paper addresses a relevant problem in multi-agent reinforcement learning. Using global dynamical information to improve decentralized actor representations is a reasonable approach, especially under partial observability. The training-only supervision is also appealing because it introduces no additional execution-time computation.

  2. The framework is clearly presented and technically coherent. The Koopman autoencoder learns predictive representations, while a simple auxiliary head transfers this information to the actor backbone. The separation between dynamics learning and policy optimization makes the method relatively easy to integrate with existing actor-critic architectures.

  3. The experimental results are promising, particularly on Ant. Table 1 reports consistent improvements across all Ant partitions, including an increase from 1,454 to 3,267 on Ant 2×4 under partial observation. The authors also provide useful ablations of auxiliary targets, autoencoder losses, and key hyperparameters.

Critical weaknesses
  1. The specific benefit of Koopman-based representation learning remains unclear. ART builds on established ideas in auxiliary representation learning and Koopman-based policy optimization. Table 2(a) shows that a fixed linear projection of the global state already achieves substantial gains. On Ant 4×2 under full observation, this baseline reaches 4,515 compared with 3,226 for MAPPO and 5,128 for ART. Similar results appear under partial observation. These results suggest that a substantial portion of the gains comes from auxiliary representation supervision. Although the prediction-loss ablations provide evidence that dynamics learning offers additional benefits, they do not establish whether the Koopman formulation is more effective than other learned predictive representations. A comparison with a similarly sized non-Koopman predictive model would help clarify this distinction.

  2. The contribution of explicit multi-agent dynamics modeling is insufficiently examined. In Section 4.1, each agent's Koopman encoder receives the same global state, distinguished by an agent identifier. Although the bilinear transition model is designed to capture joint-action interactions, the auxiliary target in Equation 11 is the current-state encoding learned under predictive objectives. The experiments do not isolate whether the improvements depend on explicit cross-agent coupling or primarily on transferring globally informative state representations. An ablation that removes cross-agent interactions from the transition model would help clarify this contribution. Further analysis of the information retained by actor features under partial observability would also be useful.

  3. The evaluation is limited in scope, and the computational cost is not fully assessed. All experiments use MAPPO on Multi-Agent MuJoCo, where agents control different joints of a single robot. Generalization to other MARL algorithms and coordination settings remains untested. Moreover, although execution is unaffected, ART requires additional autoencoder training and multi-step prediction. Reporting training time and computational overhead would help assess its practical efficiency.

Questions for authors
  1. Can the authors clarify the advantage of Koopman-based targets over generic auxiliary supervision, ideally through comparison with a non-Koopman predictive representation model?

  2. How important is explicit cross-agent coupling in the bilinear transition model? Could the authors provide a corresponding ablation or further evidence that the learned representations capture joint dynamics?

2 comments

We really appreciate that the reviewer found our framework clearly presented and technically coherent, and the experimental results promising. We address the remaining concerns below.

W1 & Q1: Koopman targets vs generic auxiliary supervision. Auxiliary supervision alone does not explain ART's gain. What matters is what the target carries. A fixed projection or an untrained encoder is a fixed function of the state and carries no information about how the system evolves. ART's target comes from a Koopman encoder/decoder trained so that its latent evolves linearly under the joint action of all agents, so it carries information about how the whole system moves. We test this with two controls. First, an untrained encoder with the same architecture as ART's encoder isolates the effect of learning the target. Second, a matched non-Koopman dynamics model keeps the same Koopman encoder/decoder and losses but replaces the bilinear transition with an MLP of the same size (63,987 vs 64,176 parameters), which isolates the effect of the Koopman form. The untrained encoder is clearly worse than ART (3,883 ± 792 vs 5,128 ± 163 on full-observation Ant 4×2, lower on all 6 seeds) and falls even below the fixed projection. The MLP model matches ART under full observation (5,206 ± 165 vs 5,128 ± 163 on Ant 4×2) and is worse under partial observation (Table G1 in our general response). The gain therefore comes from learning the target from the dynamics, and the Koopman form is as good as or better than a generic dynamics model of the same size. The results are added to the Appendix of the revised paper.

W2 & Q2 (Partial): Analysis of Multi-Agent Coupling. To isolate the effects of cross-agent coupling in the bilinear transition model, we ran two ablations of the transition model, each removing one way in which it couples the agents. (1) An additive linear model, zt+1=Kzt+Batz_{t+1} = Kz_t + Ba_t, removes the bilinear term, so the joint action can no longer change how the latent state evolves. (2) A block-diagonal KK removes the cross-agent state terms, so each agent's latent evolves only from its own latent. The results are shown in Table G2 of our general response (same paired 3 seeds).

Under partial observation, the block-diagonal model also uses per-agent encoders and removes the cross-agent bilinear terms. The matching ART model with per-agent encoders reaches 3,193 ± 793 on the same seeds, higher on all three. Additive linear and block-diagonal perform worse in the partially observable setting, where each actor sees only its own joints, the other agent's hip angles and the torso, while achieving the same result under full observability. These results indicate that the cross-agent terms of the transition model matter when each actor observes only part of the state.

Q2 (Partial): Analysis of information retained by each algorithm. Following SAC-AE (Yarats et al., Sec. 5.4), we freeze the trained actors and fit a linear map from their features to the true simulator state. Neither actor observes the full state, so this is not an exact test, and we use it only to compare ART and MAPPO trained on the same observations. ART features reconstruct the state much more closely than MAPPO features (R² 0.86 vs 0.36 on observed joints, 0.25 vs 0.14 on unobserved joints, 6 seeds). R² measures the fraction of the state's variance that the linear map explains on held-out trajectories, where 1 means perfect recovery. This shows that ART actors retain more of the robot's state and gait. The reconstructed trajectories and a detailed explanation are shown in Appendix A.6 of the revised pdf, which will be updated shortly.

W3: Extra Results. We show some further experiments that compare MA2CL and we also try ART on HAPPO. Table G4 of our general response summarizes the HAPPO results, and Table G3 the comparison with PPG and MA2CL. ART also improves HAPPO. It raises the return under partial observation on Ant 2×4 and under full observation on HalfCheetah 6×1 and Walker2d 6×1, and it matches HAPPO on HalfCheetah 3×2 (Table G4). We also compare against two auxiliary-learning baselines on Ant 2×4 under partial observation (Table G3). PPG, which regresses the actor onto the value target in a separate phase, falls below HAPPO, while ART improves HAPPO by 38%. MA2CL, which shapes the actor with an attention model over agents and a contrastive loss, reaches a higher return than ART in this setting, at 24% more training time (4.87 h vs 3.93 h per run). On matched A100 hardware, ART adds 15–29% training time over MAPPO with unchanged peak memory, and adds nothing at execution.

Hello Reviewer mW9P02,

Thank you for your valuable feedback. We hope our additional results and responses have adequately addressed your concerns. As tomorrow is the last day to engage with reviewers, we wanted to send a note to see if we have successfully addressed your concerns. We would greatly appreciate any engagement and thank you for your feedback so far.

Reviewer_JPaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies the challenge of representation learning in the context of Multi-Agent Reinforcement Learning, where the state representation can be under utilized, especially in the context of partial observable environments. This paper proposes ART that utilizes additional auxiliary loss between a Kooperman autoencoder produced representation and actor backbone to steer the actor learning. The overall methodology seems to be sound but limited in novelty. Experiments in Multi-Agent MuJoCo show improved performance.

Critical strengths
  1. This paper articulates the loss equations, pseudo-code, and learning process clearly.

  2. The proposed methodology also seems to be expandable to general reinforcement learning format. In this reviewer's opinion, this could be an advantage per se. But it is also not strongly related to the problem setting of multi-agent reinforcement learning.

Critical weaknesses
  1. The novelty of proposed methodology is limited. The concept of auxiliary loss is quite common in the field of ML and RL. The paper also use an existing autoencoder format and plug it in the auxiliary loss for actor.

  2. The experiment is very limited. Even though the paper did a comprehensive literature review and referenced multiple related works, including PPG, HAPPO, etc, only MAPPO is compared.

  3. The analysis also falls short. The paper claimed that representation auxiliary loss assisted the learning performance but never really provide further evidence to support this claim. It critiqued the direct usage of learned representation as actor input from methods like TIMAR and MAPO-LSO, but never provided evidence such as performance comparison or in-depth analysis of produced representation.

Questions for authors
  1. The proposed methodology does not seem to have a very strong relationship with the setting of MARL, except the fact that it is using MAPPO as underlying value function training and part of actor training. Can authors explain what is the uniqueness in the proposed ART that facilitate the MARL specifically? Or is ART in fact applicable to RL in general?

  2. The fixed target results being only slightly lower than ART actually appear to be concerning. The fixed target would essentially function as a innate entropy regularization for the actor loss, given the fixed target being "a static linear projection initialized with Gaussian entries and unit-norm rows". This could suggest that the randomness of learning policy is boosted during the training process and elicited more state exploration, which is known to improve RL performance. But this interpretation would contradict with the paper's proposition of representation learning's importance.

  3. Can authors provide some insight of how the representation from all compared methods differ? Since one of the highlighted claim of this paper is the importance and usage of additionally learned state representation, how the learned representation distribute is crucial for understanding the learning effect of the proposed methodology.

4 comments

(1/2)

We sincerely thank the reviewer for the positive feedback on the clarity of our method and for noting its potential beyond MARL.

W1: Novelty. Auxiliary losses and autoencoders are standard tools, and we do not claim either as our contribution. Methods such as PPG, CURL and SPR all use an auxiliary loss, and what distinguishes them is what the auxiliary loss is trained to predict. ART contributes a new target for multi-agent actors: the coordinates of a Koopman encoder/decoder in which the joint dynamics of all agents evolve linearly and are driven by the joint action through a bilinear term. As shown by our results in Section 5.2, Table 1, as well as the new results in our general response (Tables G3 and G4), this method is effective in improving performance of learned MARL policies.

Q1: ART's Relation with MARL. Although the transfer mechanism itself is general, and ART could be applied to single agent RL, two parts of ART are specific to MARL. First, it is built around CTDE. The Koopman encoder/decoder is trained centrally on the full state and the joint action, while each actor keeps acting on its local observation, so the information gap ART closes is the one created by decentralized execution. In single-agent RL, this gap requires a privileged-information setting such as sim-to-real training, whereas in cooperative MARL it is part of the paradigm. Second, the Koopman model is structured by agents. Each agent has its own block of the latent, which is the target for that agent's actor, the operator KK couples the blocks of different agents, and the bilinear term lets every agent's action change the dynamics of the joint latent. Each actor's target therefore encodes how its teammates' states and actions shape its own future. This cross-agent structure is used: with local encoders on Ant 2×4 under partial observation, removing it lowers the return from 3,193 ± 793 to 2,252 ± 106 on the same three seeds.

Q2: Innate Entropy. We disagree with the characterization of fixed-target auxiliary learning as an innate entropy regularizer. In our introduction, we cite Understanding and Preventing Capacity Loss in Reinforcement Learning by Lyle et al., who show that networks trained on changing RL targets can progressively lose feature diversity and learning capacity. They demonstrate that regressing auxiliary linear heads onto the network's own outputs at initialization can mitigate this collapse and improve performance, including on challenging sparse-reward games, without introducing a dedicated exploration mechanism. The fixed targets in our comparison employ a closely related mechanism, since both regress linear heads on the features onto a fixed, randomly constructed function of the input, suggesting that their benefits can arise from preserving representations rather than injecting randomness. Importantly, these targets do not act as an innate entropy regularizer. Once sampled, each target is a deterministic function of the state. For our Gaussian policy, the entropy of the unsquashed action distribution depends only on its separate, state-independent standard-deviation parameter, which receives no gradients from the auxiliary loss. Thus, the auxiliary objective cannot directly increase Gaussian policy entropy. It can nevertheless influence exploration indirectly by changing the actor's learned representation and resulting action distributions, which is precisely the representation-mediated effect our paper investigates.

(2/2)

W3 (Analysis) & Q3: Representation Comparison. One major difference that we would want to highlight from the mentioned papers is that we do not use attention or a communication mechanism to obtain the encodings. MA2CL uses an attention module over agents to shape the actor with a contrastive loss, although they too remove their module at deployment. The MA2E and MAPO-LSO keep their modules inside the policy at deployment. MA2E adds a Transformer masked autoencoder to every agent's network, which infers the other agents' trajectories from local observations during execution. MAPO-LSO builds its latent state from observation encoders, communication between agents and a recurrent state. TIMAR trains the observation encoder with a Transformer-based joint transition model. ART uses neither attention nor communication. Compared with them, ART trains a Koopman encoder/decoder whose bilinear operator predicts the next state in the Koopman domain. The encoder's latent of the current observation is the target, and a linear head on the actor regresses it. At deployment we remove the head and the Koopman model.

W2 & W3: Auxiliary Loss & Extra Results. We do analyze the effect of auxiliary loss in our paper, we show the effect of auxiliary loss as we increase the weight of the auxiliary loss λaux\lambda_{aux}, see Appendix A.5, Table 5. On full-observation Ant 4×2, the return rises from 3,506 ± 647 at λaux=0.01\lambda_{aux} = 0.01 to 5,128 ± 223 at 0.1 and stays flat up to 1.0 (5,150 ± 166), so the gain grows with the strength of the auxiliary signal and is not sensitive to the exact weight once it is large enough. Additionally, we also compare ART with PPG, MA2CL, and also run the mechanism with HAPPO. The results are shown in Tables G3 and G4 of our general response. ART also improves upon HAPPO as we can see from Table G4. PPG performs worse than ART in this setting while MA2CL is able to attain even higher rewards than ART (Table G3). This shows that the benefit of the Koopman target carries over to a second MARL algorithm and exceeds that of a value target, while MA2CL remains a strong alternative at a higher training cost: on the same hardware it adds 45% training time over MAPPO, compared with 17% for ART.

Hello Reviewer JPaV05,

We appreciate your feedback and hope our responses have adequately addressed your concerns. As tomorrow is the last day to engage with reviewers, we wanted to send a note to see if we have successfully addressed your concerns. We would greatly appreciate any engagement and thank you for your feedback so far.

The reviewer would like to thank authors for additional results and detailed rebuttals. However, some of the reviewer's concerns are still not addressed:

  1. | the entropy of the unsquashed action distribution depends only on its separate, state-independent standard-deviation parameter,

    • Even if only the mean value of policy distribution is updated, a static target will push it away from fitting towards the actual training target, which in terms reduces the likelihood of target action and increase the overall entropy in terms of the prediction distribution. L2 regularizer has a similar effect on supervised learning. The difference is the actor policy here will make the future collected data more diverse.
  2. | Lyle et al. .... demonstrate that regressing auxiliary linear heads onto the network's own outputs at initialization can mitigate this collapse and improve performance,...

    • Lyle et al's work actually aligns with the reviewer's point, where they use the embedding from random initialization as additional targets to work as a regularization. If you look at their experiment results in Montezuma's revenge, which is a sparse reward environment, introduction of their InFeR makes the positive return from learning episodes appearing earlier than other baselines. This suggests its impact on exploration because learning agents have to arrive at that reward state to start getting positive return in the first place.
    • However, Lyle et al's paper presented additional measurements like MSE and Feature Rank to further support their argument of InFeR learning more robust representations. The reviewer cannot find similar measurements or analysis in this ART paper, where paper's argument is learning more effective representation as well.
  3. | Representation Comparison; analyze the effect of auxiliary loss

    • While the reviewer does appreciate the additional experiment results, they do not address the reviewer's request for additional analysis on representation. To clarify, evaluation return does not directly measure the effectiveness of representation.
    • To re-iterate, the reviewer asked for the distribution comparisons between representation or embedding learned from different compared algorithms. Do some algorithm's learned representations collapse to a small cluster? How do ART's learned representations distribute?

The reviewer believes this paper has potential. However, due to unaddressed concerns, the reviewer would like to maintain the original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Extra Results

We thank all reviewers for their constructive feedback. Following their suggestions, we ran additional experiments for this rebuttal. All values are final evaluation returns, mean ± population standard deviation, and the individual responses refer to the tables below.

Table G1: Matched MLP transition (MAPPO, 6 seeds). Same Koopman encoder/decoder, losses, input, horizon and training budget as ART, with the bilinear transition replaced by an MLP g(zt,at)g(z_t, a_t) of matched size (63,987 vs 64,176 parameters).

SettingMAPPOARTMLP transition
Ant 2×4, partial obs.1,454 ± 1013,267 ± 6602,906 ± 684
Ant 4×2, full obs.3,226 ± 6485,128 ± 1635,206 ± 165

Table G2: Coupling ablations (MAPPO, same 3 seeds). The additive linear model removes the bilinear term (zt+1=Kzt+Batz_{t+1} = Kz_t + Ba_t). The block-diagonal model removes the cross-agent state terms of KK.

SettingMAPPOAdditive linearBlock-diagonal KART
Ant 2×4, partial obs.1,489 ± 522,219 ± 3322,252 ± 1062,812 ± 587
Ant 2×4, full obs.3,082 ± 5354,604 ± 8024,533 ± 7914,613 ± 745

Under partial observation, the block-diagonal model also uses per-agent encoders and removes the cross-agent bilinear terms. The matching ART model with per-agent encoders reaches 3,193 ± 793 on the same seeds.

Table G3: Auxiliary-learning methods (Ant 2×4, partial obs., same 3 seeds)

MethodReturn
MAPPO1,489 ± 52
MAPPO + MA2CL3,869 ± 286
MAPPO + ART2,812 ± 587
HAPPO2,973 ± 473
HAPPO + PPG2,051 ± 213
HAPPO + ART4,109 ± 716

Table G4: ART with HAPPO (3 seeds)

SettingHAPPOHAPPO + ART
Ant 2×4, partial obs.2,973 ± 4734,109 ± 716
HalfCheetah 6×1, full obs.4,741 ± 1436,177 ± 457
Walker2d 6×1, full obs.4,447 ± 525,227 ± 642
HalfCheetah 3×2, full obs.5,052 ± 3155,190 ± 75

Table G5: Centralized policy (full obs.). A single MAPPO agent that receives the full state and controls all joints (centralized execution) against ART with decentralized execution.

MethodReturn
Centralized MAPPO, Ant 1×8 (6 seeds)4,690 ± 350
ART, Ant 4×2 (6 seeds)5,128 ± 163
ART, Ant 8×1 (6 seeds)5,040 ± 319

Table G6: Training time per run (6 seeds each, on A100 40GB). ART does not change peak memory (9.49 vs 9.49 GiB on Ant 2×4, 9.93 vs 9.90 GiB on Ant 4×2) and adds nothing at execution.

SettingMAPPOARTOverhead
Ant 2×4, partial obs.3.36 h3.93 h+17%
Ant 4×2, partial obs.6.12 h7.01 h+15%
Ant 2×4, full obs.3.33 h3.86 h+16%
Ant 4×2, full obs.5.77 h7.18 h+24%
Ant 8×1, full obs.11.10 h14.06 h+27%
HalfCheetah 6×1, full obs.7.84 h10.10 h+29%
Walker2d 6×1, full obs.8.41 h10.05 h+20%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.