acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Representation Drift in MARL with Trust-region Autoencoders

Abstract

Vision-based multi-agent reinforcement learning (MARL) suffers from poor sample efficiency, limiting its practicality in real-world systems. Representation learning with auxiliary tasks can enhance efficiency; however, existing methods, including contrastive learning, often require the careful design of a similarity function and increase architectural complexity. In contrast, reconstruction-based methods that utilize autoencoders are simple and effective for representation learning, yet remain underexplored in MARL. We revisit this direction and identify unstable representation updates (representation drift) as a key challenge that limits the sample efficiency and stability in MARL. To address the challenge of representation drift, we propose the Multi-agent Trust Region Variational Autoencoder (MA-TRVAE), which stabilizes latent representations by regularizing updates within a trust region. Combined with Multi-agent Policy Proximal Optimization (MAPPO), MA-TRVAE improves its sample efficiency and stability in vision-based multi-agent control tasks. Experiments demonstrate that this simple approach is competitive with state-of-the-art MARL methods, while being more computationally efficient. Furthermore, we show that our method can scale up to more agents with only slight performance degradation.

Primary area:
reinforcement learning
Keywords:
Representation Learning, RL, Multi-agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6CJFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Agent Trust Region Variational Autoencoder (MA-TRVAE), a representation learning framework designed to improve sample efficiency and training stability in vision-based multi-agent reinforcement learning (MARL). Specifically, the authors identify representation drift as a potential source of instability when jointly training variational autoencoders (VAEs) and MARL policies. To address this issue, MA-TRVAE introduces a trust-region regularization term that constrains the KL divergence between the latent distributions produced by the current and previous encoders. The proposed method is integrated with MAPPO to stabilize representation learning while preserving the expressiveness of the latent space. Experiments on Multi-Agent Quadcopter Control, Google Research Football, and Multi-Agent MuJoCo demonstrate improvements in sample efficiency and training stability over several baselines.

Critical strengths
  1. This paper investigates representation drift in vision-based MARL, which is a meaningful problem since continuously changing latent representations may negatively affect policy learning and sample efficiency.

  2. The proposed MA-TRVAE introduces a simple KL regularization term to constrain representation changes between consecutive encoder updates, which can be easily integrated into existing MARL algorithms without substantial architectural modifications.

  3. The authors provide empirical analyses of representation drift under different VAE regularization strengths, demonstrating the relationship between latent representation stability and downstream MARL performance.

  4. The experimental results show that MA-TRVAE generally improves the sample efficiency and training stability of MAPPO-based methods. The additional experiments on scalability and computational efficiency further demonstrate the practical potential of the proposed approach.

Critical weaknesses
  1. The proposed trust-region regularization essentially introduces a KL divergence constraint between the outputs of the current and previous encoders, which is closely related to existing consistency regularization and target encoder approaches. Moreover, the authors acknowledge that using an EMA target encoder achieves similar performance in Appendix G, making the methodological distinction from existing representation stabilization techniques less convincing and the contribution incremental.

  2. Although the experiments show that stronger regularization reduces posterior distance, this does not necessarily lead to better policy performance. In fact, Fig. 8 shows that excessive regularization can reduce the final return. Therefore, the empirical results do not fully explain when or to what extent mitigating representation drift benefits MARL training.

  3. MA-TRVAE does not explicitly address representation learning challenges specific to multi-agent systems. MA-TRVAE applies the same reconstruction and representation consistency objectives to individual agents' observations without explicitly modeling inter-agent interactions or the state info. Moreover, the single-agent experiments in Appendix H demonstrate similar benefits, raising questions about what makes the proposed approach particularly suitable for MARL beyond directly applying existing single-agent representation learning techniques.

  4. The main vision-based experiments primarily focus on two quadcopter control tasks, while Google Research Football uses shared global visual observations and Multi-Agent MuJoCo relies on state-based observations. Furthermore, MA-TRVAE underperforms MAT in the more challenging DYN control settings, and its performance degrades substantially when scaling to 10 agents. These results limit the generality of the claimed performance and scalability advantages.

Questions for authors
  1. Please clarify the key methodological differences between MA-TRVAE and existing representation consistency regularization methods, particularly EMA-based target encoders. Given the similar performance reported in Appendix G, what is the specific advantage of using the previous encoder? (i.e., the novelty of this work, See Weakness 1)

  2. Why should reducing representation drift necessarily improve downstream MARL performance? Since Fig. 8 suggests a trade-off between representation stability and policy performance, can the authors provide a more detailed analysis of this relationship? (See Weakness 2)

  3. What are the specific challenges of representation drift in MARL compared with single-agent RL? Can the authors provide empirical evidence demonstrating that the proposed regularization is particularly beneficial for multi-agent learning rather than being a general representation stabilization technique? (See Weakness 3)

  4. Can the authors provide additional evaluations on more challenging vision-based MARL benchmarks and larger agent populations? In particular, why does MA-TRVAE underperform MAT in the DYN control settings, and how can its scalability be improved beyond 7 agents? (See Weakness 4)

Considering its limited methodological novelty in comparison to existing representation learning techniques, the reviewer believes that this paper currently falls below the acceptance threshold.

3 comments

We thank the reviewer for the evaluation. We address the raised questions below.

Q1

We agree the regularizer is closely related to consistency regularization; Sec. 5.3 frames it as a special case of a Mean Teacher [1]. The KL penalty itself is not our claimed novelty. Our contribution is identifying representation drift as a key failure mode of reconstruction-based MARL and showing that regulating encoder updates resolves it simply and scalably. App. G supports this framing: two different ways of regulating drift give similar gains, so the drift itself, not the specific regularizer, is what matters.

  • Differences from target-encoder methods. In Mean Teacher, the target network provides the learning signal: the student matches the teacher’s outputs on perturbed inputs. In MA-TRVAE, reconstruction provides the learning signal. The trust-region term only regularizes the size of each encoder update on identical inputs, so it acts as a step-size regularization (Eq. 5), not a prediction target. It also constrains the full posterior, including its variance, rather than a deterministic representation as used in most consistency regularization techniques.
  • Advantage of using previous encoder.
    • The KL to the previous encoder is exactly the per-update drift the actor and critic face. An EMA target regularizes this drift only indirectly.
    • It needs no EMA averaging operation and decay hyperparameter, which would interact with β2\beta_2.
    • In App. G, it shows lower variance across seeds than the EMA variant at similar returns (Fig. 17a).

Q2

We agree, and we do not claim that minimizing drift is monotonically beneficial (see Sec. 5.3 and App. H.2.2). Our claim is narrower: uncontrolled drift harms MARL training, and regularizing it, rather than eliminating it, improves stability and sample efficiency. Every trust-region method faces this trade-off. For example, in PPO, an overly small clipping range slows learning, while a large one hurts performance (Table 1 in [2]); the regularization strength must be selected appropriately. Likewise, an overly large β2\beta_2 prevents the encoder from improving, while β2=0\beta_2 = 0 allows uncontrolled drift, and a moderate β2\beta_2 balances the two.

To be more precise about why reducing representation drift improves downstream MARL sample efficiency and stability, consider how MARL updates its policy after encoder updates. The encoder update changes the latent of the same observation; the actor must re-map it to the same action, and the critic’s value estimates become stale. Part of each update is then spent compensating for representation change rather than improving the policy. If the representation drift is reduced, each policy update can be optimized more for a better policy.

However, if the regularization is too strong, the encoder is forced to stay close to its earlier, less-trained version. As the policy improves and visits new states, the encoder needs to change to encode this new information, and a strong penalty prevents this. The latent therefore stays under-informative, which explains the drop in final return for large β2\beta_2 in Fig. 8.

Q3

We thank the reviewer for raising this. We agree that the trust-region objective is not tied to multi-agent settings, and we included App. H to show exactly that. Our claim is that MARL amplifies the problem our MA-TRVAE addresses: representation drift.

  • MARL prefers weak representation regularization that increases drift. In Fig. 1a, β\beta-VAEs with β∈{1,0.1}\beta \in \{1, 0.1\} perform worse than MAPPO without representation learning, and good performance requires β\beta as small as 1e−71\mathrm{e}{-7}. [3] do not observe such a gap in single-agent RL. We attribute this to heterogeneity of observations: one shared encoder must compress NN different ego-centric views, which a strongly regularized latent cannot do. Weak regularization, however, is exactly what causes drift (Fig. 1b). The trust region resolves this trade-off, keeping the expressiveness of a small β1\beta_1 with the stability of a constrained latent.

  • Drift compounds through the centralized critic and grows with the number of agents. The encoder is shared, trained on observations pooled across agents, and updated through the centralized critic, whose input is [z1,…,zN][z^1, \dots, z^N]. Because the posterior factorizes across agents, the drift in the critic’s input is KL(qϕ(z1:N∣o1:N) ∥ qϕold(z1:N∣o1:N))=∑i=1NKL(qϕ(zi∣oi) ∥ qϕold(zi∣oi)).\mathrm{KL}\big(q_\phi(z^{1:N} \mid o^{1:N}) \,\|\, q_{\phi_{\text{old}}}(z^{1:N} \mid o^{1:N})\big) = \textstyle\sum_{i=1}^{N} \mathrm{KL}\big(q_\phi(z^i \mid o^i) \,\|\, q_{\phi_{\text{old}}}(z^i \mid o^i)\big). Our trust-region penalty minimizes this drift on the joint representation the critic uses. Without it, critic-input drift grows linearly with NN and corrupts every agent’s advantage estimates at once. This has no single-agent analog and matches our scaling results. From 4 to 7 agents, MA-TRVAE degrades only slightly, while the baselines drop in performance (Fig. 5) and the unregularized VAE/AE variants become less stable (Fig. 15).

  • We deliberately do not model inter-agent interactions explicitly. Inter-agent information already reaches the encoder through the critic’s gradient, and under CTDE, actors must act on local latents at execution anyway. Attention-based alternatives (MA2CL, MAT) do bring more inter-agent information in the latent representation but need about 20% more training time (Figs. 6, 14) and degrade more as NN grows (Fig. 5). The trust-region term is also complementary to such modules and can be added to any online-updated encoder.

Q4

We agree that the vision-based evaluation is limited, as stated in our Limitations. Benchmarks with per-agent ego-centric pixel observations in a Dec-POMDP are rare. That is why our primary testbed is MAQC, which is also the main benchmark of MA2CL. Within MAQC, we cover two tasks, three action settings (PID, DYN, RPM), 4–10 agents, and noisy observations. GRF, MA-MuJoCo and eight single-agent DMC tasks complement these results.

  • Why does MA-TRVAE underperform MAT in the DYN control settings? Among MAPPO-based methods, which isolate the effect of representation learning, MA-TRVAE is best in every MAQC setting (Table 1). MAT changes the policy architecture itself: it generates actions autoregressively, conditioning each agent’s action on those of the preceding agents. We believe this explains the gap in Flock-DYN. There, each drone tracks the preceding drone’s altitude and, without PID stabilization, must coordinate low-level thrust with it, which is a structure that matches MAT’s sequential decoding. In Leader-DYN, where all agents track a common leader, the two methods are within one standard deviation (−14.09 ±\pm 1.49 vs. −12.20 ±\pm 2.56).

How can scalability be improved beyond 7 agents? At 10 agents, performance drops for all methods, which points to the underlying MAPPO setup rather than to the trust region. We see two likely causes:

  • Representation capacity. The centralized critic takes NN concatenated latents but keeps a fixed size of 64 (Table 2). An increased latent size may improve performance.
  • Drift increasing. The representation drift is the sum of per-agent drifts. With a fixed β2\beta_2, the effective trust-region regularization on the representation therefore weakens as NN grows. Increasing β2\beta_2 with NN would compensate for this.

[1] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).

[2] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

[3] Yarats, Denis, et al. "Improving sample efficiency in model-free reinforcement learning from images." Proceedings of the aaai conference on artificial intelligence. Vol. 35. No. 12. 2021.

Reviewer_6CJFOpenReview ↗

Reply to the authors' response

Thank you for the authors' response. However, I still believe that the novelty of this work is limited. Therefore, I will maintain my original score.

Reviewer_MtYcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies representation learning for vision-based multi-agent reinforcement learning and argues that encoder updates can cause representation drift that harms downstream MAPPO training. It first combines a beta-VAE with MAPPO, then proposes MA-TRVAE, which penalizes the KL divergence between the current encoder posterior and that of a previous encoder. The method is evaluated on multi-agent quadcopter control, Multi-Agent MuJoCo, Google Research Football, and several single-agent control tasks. The reported results suggest that reconstruction-based representations can improve MAPPO and that the proposed consistency penalty often reduces the authors' posterior-distance metric and cross-seed performance variance.

Critical strengths
  1. The paper studies a relevant and underexplored practical problem. Vision-based MARL is sample-inefficient, and instability in a jointly trained shared encoder is a plausible source of difficulty. The proposed old-encoder consistency penalty is simple and potentially useful.

  2. The empirical scope is broader than a single benchmark. The paper includes visual MAQC, state-based Multi-Agent MuJoCo, Google Research Football, scalability tests, wall-clock measurements, noise experiments, an EMA comparison, and single-agent controls. This breadth makes the observed regularization effect worth investigating further.

Critical weaknesses
  1. The paper gives two materially different definitions of the trained algorithm. Section 3.1 (p. 4) and the Figure 2 caption (p. 6) state that the actor, critic, and TRVAE losses all backpropagate into the shared encoder. Algorithm 1 (p. 19, lines 987–988), however, updates the actor and critic with PPO and then updates the encoder/decoder only with the TRVAE loss. Whether policy/value gradients enter the encoder directly changes both the representation-drift process and the interpretation of the regularizer. Until the actual update rule is identified, the method and its results are not reproducible. This is compounded by contradictory MAQC input shapes—64x64x4 on p. 3 and 64x48x4 on p. 6—and by a Dec-POMDP formalization that omits the hidden state and observation model.

  2. The paper's own experiments do not identify drift as the cause of poor sample efficiency. Figure 1 (p. 4) shows that reducing beta increases posterior distance while improving reward, and Figure 8 (p. 9) shows that stronger trust-region regularization reduces drift but worsens performance. Figure 7 further suggests that the main mean-return improvement comes from adding reconstruction, while the trust penalty mainly changes cross-seed variance. Because beta2 simultaneously changes encoder update magnitude, information retention, and optimization conditioning, these plots do not establish the claimed chain "drift reduction causes better sample efficiency." The paper also uses two non-equivalent drift protocols without separating their interpretation: the main analysis follows a fixed observation (p. 5), whereas Appendix H.2 computes KL on the current observations (pp. 23–24), whose distribution changes with the policy. The latter quantity therefore conflates encoder drift with on-policy distribution shift and cannot be compared directly with the fixed-observation measurement.

Questions for authors
  1. Which losses actually update the encoder in the reported experiments: only the TRVAE loss, as indicated by Algorithm 1, or the actor, critic, and TRVAE losses, as stated in Section 3.1 and Figure 2? Please clarify the optimized parameter sets, the gradient flow into the shared encoder, and the update order of PPO and TRVAE. This distinction materially changes both the source of representation drift and the interpretation of the proposed regularizer.

  2. Was posterior distance computed on one fixed observation across encoder updates or on the current on-policy observations? Please provide the exact protocol and a controlled experiment that varies the consistency penalty while matching reconstruction, encoder update magnitude, and observation distribution. Under such a controlled intervention, does reducing posterior distance actually accelerate learning, or does it mainly smooth the learning curve and reduce return variability?

  3. How is the "scaled episode reward" in Figure 5 defined when the number of agents changes? Since Appendix C.1 defines team reward as a sum over agents, the scalability claim is not interpretable without the normalization.

1 comment

We thank the reviewer for the evaluation. We address the raised questions below.

Q1: Which losses actually update the encoder?

The actor, critic and TRVAE losses are all used to update the shared encoder. We realize we made a mistake in Algorithm 1: we did not include the encoder parameters in the backpropagation of the actor and critic losses. We apologize for the ambiguity in the report and will correct it in the final version. We would like to stress that the mistake is only in the pseudocode, not in the actual code used to run the experiments.

Q2: Was posterior distance computed on one fixed observation across encoder updates or on the current on-policy observations?

The posterior distance is computed on the fixed observation across encoder updates. The protocol is executed as below:

  1. Sample a batch of observations $\mathbf{o} $ from a rollout collected through a random policy.
  2. For every k training steps, we calculate the posterior distance by DKL(qϕ(z∣o)∣∣qϕold(z∣o))D_{\mathrm{KL}}(q_{\phi}(\mathbf{z}\vert \mathbf{o}) \vert \vert q_{\phi_{old}}(\mathbf{z}\vert \mathbf{o}))

where qϕq_{\phi} is the current encoder and qϕoldq_{\phi_{old}} is the saved previous encoder.

Fig. 3 (page 7) shows the posterior distance using different β2\beta_2 values, i.e., varying the consistency penalty, with all other settings matching, including the same reconstruction penalty (β1\beta_1), encoder update magnitude (encoder's learning rate) and observation distribution (as we explained in the protocol). We observe that the posterior distance decreases when we introduce the consistency penalty, where MAPPO + VAE equals MAPPO + MA-TRVAE with β2=0\beta_2 = 0. However, we also observe that at β2=1e−5\beta_2 = 1\mathrm{e}{-5}, the posterior distance is greatly reduced, and a stronger penalty reduces it even more. This can lead to representation collapse, i.e., the representation barely changes from the randomly initialized encoder when we use a large consistency penalty. The downstream policy learned from such a collapsed representation therefore performs worse in the MARL tasks, which explains the seemingly contradictory fact from Fig. 8 that stronger trust-region regularization reduces drift but worsens performance.

We therefore do not claim that "reducing representation drift improves the downstream MARL performance monotonically" but a narrower one: uncontrolled drift harms MARL training, and regularizing it, rather than eliminating it, improves stability and sample efficiency.

Q3: How is the "scaled episode reward" in Figure 5 defined when the number of agents changes?

We thank the reviewer for raising this question. We calculate the reward exactly as defined in Appendix C.1, so it is not normalized. We will replot the curve with a normalized reward in the final version. Nevertheless, we want to point out that more agents add more penalty to the reward according to Appendix C.1; therefore, it is more difficult to maintain the same reward with more agents, which is one reason the episode reward drops significantly when we scale to 10 agents. Note that our method maintains the same episode reward from 4 to 7 agents, while the others do not (Fig. 5).

W1: input shape clarification and incomplete Dec-POMDP definition

We apologize for the different input shapes. The correct one is 64 x 48 x 4. We will correct it in the final version. For the incomplete Dec-POMDP definition, we will add both the state and observation models to reduce ambiguity.\

W2: two non-equivalent drift protocols

We apologize for the mistake. For the single-agent experiment, we plot the posterior distance with both the current observation and the fixed observation as input to the encoders. We will replace the current plot with the one that uses a fixed observation as input to make the metric consistent.

Reviewer_RwCMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper at visual multi-agent RL and notices that the small-β VAE trick that works in single-agent RL actually makes MAPPO worse. They blame the encoder drifting between policy updates, the same observation ends up with a different latent after each update, and fix it by adding a PPO-style KL penalty that pins the new encoder close to the old one. The result (MA-TRVAE) is basically MAPPO plus one extra loss term, tested mostly on a custom quadcopter env, plus a couple of small google research football task and state-based Multi-Agent MuJoco.

Critical strengths

The drift diagnosis is a genuine observation: small β, which single-agent visual RL needs, destabilizes the shared encoder under multi-agent alternated updates, and the posterior-KL-over-time plot makes it visible.

Critical weaknesses

The method is just PPO's KL penalty applied to the encoder, and the authors admit it's basically the same as an EMA/mean-teacher. No new mechanism, no theorem, and the obvious control—just running an EMA encoder—is buried in the appendix instead of the main comparison.

Questions for authors

How does MA-TRVAE compare to (a) an EMA encoder with the same consistency objective and (b) a frozen pretrained visual encoder (R3M/VIP) on the same MAQC/GRF tasks? These are the two most direct baselines for your claim.

1 comment

We thank the reviewer for the positive comment. We address the raised questions below.

How does MA-TRVAE compare to an EMA encoder with the same consistency objective?

In Fig. 17, the EMA encoder variant already uses the consistency regularization between the current encoder and the target encoder. The only difference between the two methods is the target encoder: MAPPO + MA-TRVAE (previous encoder) uses the previous encoder as the target encoder, while MAPPO + MA-TRVAE (EMA encoder) uses the EMA encoder as the target encoder. Both of the methods are effective in reducing representation drift (Fig. 17 b) and show similar improved performance for downstream MARL tasks (Fig. 17 a), indicating that mitigating representation drift is an effective way to improve downstream MARL.

How does MA-TRVAE compare to a frozen pretrained visual encoder (R3M/VIP) on the same MAQC/GRF tasks?

We thank the reviewer for proposing this experiment. This is an interesting experiment, and we will include it in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.