- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Multi-Agent Trust Region Variational Autoencoder (MA-TRVAE), a representation learning framework designed to improve sample efficiency and training stability in vision-based multi-agent reinforcement learning (MARL). Specifically, the authors identify representation drift as a potential source of instability when jointly training variational autoencoders (VAEs) and MARL policies. To address this issue, MA-TRVAE introduces a trust-region regularization term that constrains the KL divergence between the latent distributions produced by the current and previous encoders. The proposed method is integrated with MAPPO to stabilize representation learning while preserving the expressiveness of the latent space. Experiments on Multi-Agent Quadcopter Control, Google Research Football, and Multi-Agent MuJoCo demonstrate improvements in sample efficiency and training stability over several baselines.
Critical strengths
-
This paper investigates representation drift in vision-based MARL, which is a meaningful problem since continuously changing latent representations may negatively affect policy learning and sample efficiency.
-
The proposed MA-TRVAE introduces a simple KL regularization term to constrain representation changes between consecutive encoder updates, which can be easily integrated into existing MARL algorithms without substantial architectural modifications.
-
The authors provide empirical analyses of representation drift under different VAE regularization strengths, demonstrating the relationship between latent representation stability and downstream MARL performance.
-
The experimental results show that MA-TRVAE generally improves the sample efficiency and training stability of MAPPO-based methods. The additional experiments on scalability and computational efficiency further demonstrate the practical potential of the proposed approach.
Critical weaknesses
-
The proposed trust-region regularization essentially introduces a KL divergence constraint between the outputs of the current and previous encoders, which is closely related to existing consistency regularization and target encoder approaches. Moreover, the authors acknowledge that using an EMA target encoder achieves similar performance in Appendix G, making the methodological distinction from existing representation stabilization techniques less convincing and the contribution incremental.
-
Although the experiments show that stronger regularization reduces posterior distance, this does not necessarily lead to better policy performance. In fact, Fig. 8 shows that excessive regularization can reduce the final return. Therefore, the empirical results do not fully explain when or to what extent mitigating representation drift benefits MARL training.
-
MA-TRVAE does not explicitly address representation learning challenges specific to multi-agent systems. MA-TRVAE applies the same reconstruction and representation consistency objectives to individual agents' observations without explicitly modeling inter-agent interactions or the state info. Moreover, the single-agent experiments in Appendix H demonstrate similar benefits, raising questions about what makes the proposed approach particularly suitable for MARL beyond directly applying existing single-agent representation learning techniques.
-
The main vision-based experiments primarily focus on two quadcopter control tasks, while Google Research Football uses shared global visual observations and Multi-Agent MuJoCo relies on state-based observations. Furthermore, MA-TRVAE underperforms MAT in the more challenging DYN control settings, and its performance degrades substantially when scaling to 10 agents. These results limit the generality of the claimed performance and scalability advantages.
Questions for authors
-
Please clarify the key methodological differences between MA-TRVAE and existing representation consistency regularization methods, particularly EMA-based target encoders. Given the similar performance reported in Appendix G, what is the specific advantage of using the previous encoder? (i.e., the novelty of this work, See Weakness 1)
-
Why should reducing representation drift necessarily improve downstream MARL performance? Since Fig. 8 suggests a trade-off between representation stability and policy performance, can the authors provide a more detailed analysis of this relationship? (See Weakness 2)
-
What are the specific challenges of representation drift in MARL compared with single-agent RL? Can the authors provide empirical evidence demonstrating that the proposed regularization is particularly beneficial for multi-agent learning rather than being a general representation stabilization technique? (See Weakness 3)
-
Can the authors provide additional evaluations on more challenging vision-based MARL benchmarks and larger agent populations? In particular, why does MA-TRVAE underperform MAT in the DYN control settings, and how can its scalability be improved beyond 7 agents? (See Weakness 4)
Considering its limited methodological novelty in comparison to existing representation learning techniques, the reviewer believes that this paper currently falls below the acceptance threshold.
3 comments
Q3
We thank the reviewer for raising this. We agree that the trust-region objective is not tied to multi-agent settings, and we included App. H to show exactly that. Our claim is that MARL amplifies the problem our MA-TRVAE addresses: representation drift.
-
MARL prefers weak representation regularization that increases drift. In Fig. 1a, -VAEs with perform worse than MAPPO without representation learning, and good performance requires as small as . [3] do not observe such a gap in single-agent RL. We attribute this to heterogeneity of observations: one shared encoder must compress different ego-centric views, which a strongly regularized latent cannot do. Weak regularization, however, is exactly what causes drift (Fig. 1b). The trust region resolves this trade-off, keeping the expressiveness of a small with the stability of a constrained latent.
-
Drift compounds through the centralized critic and grows with the number of agents. The encoder is shared, trained on observations pooled across agents, and updated through the centralized critic, whose input is . Because the posterior factorizes across agents, the drift in the critic’s input is Our trust-region penalty minimizes this drift on the joint representation the critic uses. Without it, critic-input drift grows linearly with and corrupts every agent’s advantage estimates at once. This has no single-agent analog and matches our scaling results. From 4 to 7 agents, MA-TRVAE degrades only slightly, while the baselines drop in performance (Fig. 5) and the unregularized VAE/AE variants become less stable (Fig. 15).
-
We deliberately do not model inter-agent interactions explicitly. Inter-agent information already reaches the encoder through the critic’s gradient, and under CTDE, actors must act on local latents at execution anyway. Attention-based alternatives (MA2CL, MAT) do bring more inter-agent information in the latent representation but need about 20% more training time (Figs. 6, 14) and degrade more as grows (Fig. 5). The trust-region term is also complementary to such modules and can be added to any online-updated encoder.
Q4
We agree that the vision-based evaluation is limited, as stated in our Limitations. Benchmarks with per-agent ego-centric pixel observations in a Dec-POMDP are rare. That is why our primary testbed is MAQC, which is also the main benchmark of MA2CL. Within MAQC, we cover two tasks, three action settings (PID, DYN, RPM), 4–10 agents, and noisy observations. GRF, MA-MuJoCo and eight single-agent DMC tasks complement these results.
- Why does MA-TRVAE underperform MAT in the DYN control settings? Among MAPPO-based methods, which isolate the effect of representation learning, MA-TRVAE is best in every MAQC setting (Table 1). MAT changes the policy architecture itself: it generates actions autoregressively, conditioning each agent’s action on those of the preceding agents. We believe this explains the gap in Flock-DYN. There, each drone tracks the preceding drone’s altitude and, without PID stabilization, must coordinate low-level thrust with it, which is a structure that matches MAT’s sequential decoding. In Leader-DYN, where all agents track a common leader, the two methods are within one standard deviation (−14.09 1.49 vs. −12.20 2.56).
How can scalability be improved beyond 7 agents? At 10 agents, performance drops for all methods, which points to the underlying MAPPO setup rather than to the trust region. We see two likely causes:
- Representation capacity. The centralized critic takes concatenated latents but keeps a fixed size of 64 (Table 2). An increased latent size may improve performance.
- Drift increasing. The representation drift is the sum of per-agent drifts. With a fixed , the effective trust-region regularization on the representation therefore weakens as grows. Increasing with would compensate for this.
[1] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).
[2] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the aaai conference on artificial intelligence. Vol. 35. No. 12. 2021.
Reply to the authors' response
Thank you for the authors' response. However, I still believe that the novelty of this work is limited. Therefore, I will maintain my original score.
We thank the reviewer for the evaluation. We address the raised questions below.
Q1
We agree the regularizer is closely related to consistency regularization; Sec. 5.3 frames it as a special case of a Mean Teacher [1]. The KL penalty itself is not our claimed novelty. Our contribution is identifying representation drift as a key failure mode of reconstruction-based MARL and showing that regulating encoder updates resolves it simply and scalably. App. G supports this framing: two different ways of regulating drift give similar gains, so the drift itself, not the specific regularizer, is what matters.
Q2
We agree, and we do not claim that minimizing drift is monotonically beneficial (see Sec. 5.3 and App. H.2.2). Our claim is narrower: uncontrolled drift harms MARL training, and regularizing it, rather than eliminating it, improves stability and sample efficiency. Every trust-region method faces this trade-off. For example, in PPO, an overly small clipping range slows learning, while a large one hurts performance (Table 1 in [2]); the regularization strength must be selected appropriately. Likewise, an overly large β2 prevents the encoder from improving, while β2=0 allows uncontrolled drift, and a moderate β2 balances the two.
To be more precise about why reducing representation drift improves downstream MARL sample efficiency and stability, consider how MARL updates its policy after encoder updates. The encoder update changes the latent of the same observation; the actor must re-map it to the same action, and the critic’s value estimates become stale. Part of each update is then spent compensating for representation change rather than improving the policy. If the representation drift is reduced, each policy update can be optimized more for a better policy.
However, if the regularization is too strong, the encoder is forced to stay close to its earlier, less-trained version. As the policy improves and visits new states, the encoder needs to change to encode this new information, and a strong penalty prevents this. The latent therefore stays under-informative, which explains the drop in final return for large β2 in Fig. 8.