acceptodds
Under review as a conference paper at ICLR 2027

Multi-Agent Information-Guided Policy Optimization: An Information-Theoretic Perspective

Abstract

In multi-agent reinforcement learning (MARL), partial observability limits the ability of decentralized agents to leverage global information, motivating widely used frameworks such as Centralized Training with Decentralized Execution (CTDE) and more recently, Multi-Agent Guided Policy Optimization (MAGPO). However, MAGPO relies solely on policy alignment and fails to explicitly align decision-relevant information between the guider and learner, limiting decentralized policy learning. In this work, we revisit the imitation gap between guider and learner from an information-theoretic perspective, and show that it fundamentally arises from an information gap—a mismatch in decision-relevant information between centralized and decentralized policies. To mitigate this gap, we propose Multi-Agent Information-Guided Policy Optimization (MAIGPO), which employs the information bottleneck principle to extract decision-relevant representations and mutual information maximization to align them between the guider and learners through bidirectional information flow. By encouraging learners to capture the guider’s decision rationale and constraining the guider to learner-accessible information, MAIGPO significantly improves decentralized policy learning and outperforms state-of-the-art methods on standard benchmarks.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Information Bottleneck, Mutual Information, Information Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8RPuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MAIGPO, which views the guider-learner imitation gap in CTDS as an information-theoretic mismatch between the decision-relevant information available to the centralized guider and the decentralized learner. Under an idealized assumption, it shows that reducing this gap amounts to maximizing the mutual information between their latent representations. To achieve this, MAIGPO augments MAGPO’s action-level imitation objective with a variational information bottleneck and a bidirectional contrastive alignment loss based on InfoNCE. Experiments across 23 tasks in RobotWarehouse and OvercookedV2 show that MAIGPO outperforms CTDE, CTDS, and CTCE baselines.

Critical strengths
  1. Principled derivation of the method. The paper derives that closing the information gap reduces to maximizing the shared information between guider and learner representations, giving it formal justification.

  2. Empirical validation. 23 tasks across two benchmark suites, comparisons against CTDE/CTCE/CTDS baselines, proper statistical reporting (IQM, bootstrapped CIs, probability-of-improvement), plus additional results on MPE/SMAX.

  3. Ablations. The authors isolate the contribution of the information bottleneck vs. the alignment loss, and specifically tests whether gains grow with the information gap itself, not just reporting aggregate performance.

Critical weaknesses
  1. The motivating example lacks detail and clarity. Figure 1 illustrates MAIGPO’s architecture rather than the failure it aims to address. Given that the information-gap interpretation is one of the paper’s central novelties and pivotal to the rest of the work, the authors should provide a concrete example or evidence showing that this gap exists and explaining why MAGPO’s action-level alignment fails to address it.

  2. Unclear and potentially insufficient seed count. The paper states each algorithm is trained with “1–5 seeds per task” but doesn’t specify how many seeds were used for which tasks. Results based on 1–2 seeds are not statistically meaningful [1]; they can’t support the confidence intervals reported (Section 5.1) and make it impossible to assess whether reported differences between MAIGPO and baselines reflect genuine improvement or seed variance, particularly on the 23 tasks evaluated. This ambiguity should be resolved with a per-task seed count, and any task run with fewer than ~3 seeds should be flagged or excluded from headline comparisons.

  3. Unverified link between task difficulty and the information gap. Section 5.2 orders tasks by an assumed increasing information gap (based on map size/agent count/queue size) and shows MAIGPO’s advantage over MAGPO grows accordingly, yet the gap itself is never directly measured, only inferred from task structure. This is consistent with the paper’s hypothesis but doesn’t confirm it, since harder tasks could favor MAIGPO for reasons unrelated to the proposed mechanism.

I would be happy to raise my score if the authors can adequately address those issues.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.

Questions for authors
  1. How many seeds were used per task specifically, and were any of the headline results (Figures 3–4) based on just 1–2 seeds?

  2. Given that βg\beta_g, βl\beta_l, λg\lambda_g, λl\lambda_l, and τ\tau are tuned separately per task (Appendix F.1), how sensitive is MAIGPO's advantage over MAGPO to this tuning budget — would the gains hold with a single shared configuration across tasks, or only under per-task search?

7 comments
AuthorsOpenReview ↗

Thank you for your positive assessment and constructive feedback

We appreciate the reviewers' careful evaluation and valuable feedback. The comments have helped us refine our arguments and better communicate the contributions of our work. We respond to each question and concern below.

[Weakness 1] The motivating example lacks detail and clarity.

Figure 1 illustrates the motivation behind MAIGPO: the guider–learner imitation gap reflects a mismatch in decision-relevant information. For example, consider the first agent in the highlighted box on the right of Fig. 1: the guider selects the upper item based on global information, allowing the other agent to collect the item nearest to it, whereas the learner selects the lower item because of its limited observations.

MAGPO’s policy alignment matches the final action distributions, influencing hidden representations only indirectly. We do not claim that action-level alignment fails to address this mismatch, but that it may be insufficient. We therefore introduce information alignment to explicitly promote decision-relevant information sharing between the guider and learner, complementing the existing policy-level constraints. Extensive experiments further demonstrate the performance gains achieved by our method.

Additionally, we use the guider–learner return difference (Figure 6) as a proxy for the information gap I(Ztg;At)−I(Ztl;At)I(Z_t^g;A_t)-I(Z_t^l;A_t). The figure shows that MAIGPO generally achieves a smaller return gap and higher performance than MAGPO, particularly on forced_coord. These results are consistent with our motivation and theoretical perspective.

[Weakness 2] Unclear and potentially insufficient seed count.

We apologize for the ambiguity. All algorithms on all tasks were trained with five independent random seeds: 1, 2, 3, 4, and 5. “1–5 seeds” was intended to denote these seed values, not a varying seed count. We will correct this wording in Section 5.1. Additionally, as described in Appendix E.1, we set os.environ["XLA_FLAGS"] = "--xla_gpu_deterministic_ops=true" in all experiments to enable deterministic GPU operations and improve reproducibility.

[Weakness 3] Unverified link between task difficulty and the information gap.

Our ordering reflects structural differences between guider and learner inputs, rather than task difficulty alone. With a fixed local sensing range, larger maps and more agents can increase the potentially relevant information unavailable to individual learners. Smaller request queues require tighter task allocation to avoid competing for the same target; the guider can coordinate these assignments using the global state and preceding agents’ actions, whereas learners lack direct access to this information.

These factors motivate the ordering as an approximate proxy for information asymmetry. The increasing advantage of MAIGPO over MAGPO further suggests that MAIGPO is better suited to settings with greater information asymmetry.

We interpret Fig. 8 as supporting evidence consistent with our hypothesis, rather than direct confirmation of the mechanism. We will revise the section and caption to clarify that the ordering serves as a structural proxy for information asymmetry.

[Q 1] How many seeds were used per task specifically, and were any of the headline results (Figures 3–4) based on just 1–2 seeds?

All experiments across all tasks and algorithms use the same five random seeds (1, 2, 3, 4, and 5) to ensure fair comparisons.

[Q 2] how sensitive is MAIGPO's advantage over MAGPO to this tuning budget — would the gains hold with a single shared configuration across tasks, or only under per-task search?

Our hyperparameter tuning procedure largely follows MAGPO, which uses Optuna-TPE with 40 trials per task. Similarly, MAIGPO uses 10 trials to tune the IB-related hyperparameters, followed by 30 trials to tune the remaining hyperparameters. We also adopt MAGPO's reported optimal hyperparameters for the underlying algorithm. Our results demonstrate that MAIGPO outperforms MAGPO under task-specific tuning, suggesting that the improvements are not merely attributable to a different tuning procedure.

Given the differences in task characteristics, a single shared configuration may not fully preserve the gains achieved through task-specific tuning. Following MAGPO, we adopt task-specific optimized hyperparameters. We will release the full parameter configurations for all tasks with our open-source code.

Dear authors, Thank you for the detailed response and for the effort you put into addressing my concerns. I appreciate the new results and the clarification regarding the seed count.

One concern remains. The motivating example in the introduction (Fig. 1) sets the stage for the entire paper, so it should be supported by evidence that the gap exists, in terms of either the guide–learner return gap or the information gap. Could the new estimator you proposed be applied to this example, for instance, by measuring the estimated gap at states matching the Fig. 1 scenario for MAGPO and MAIGPO?

Since the information-gap perspective is the paper’s central novelty and the main motivation for MAIGPO, the example illustrating it should be supported by evidence. The new evaluation results (Fig. 5) do this well; the motivating example in the introduction deserves similar support.

Dear Reviewer,

Thank you very much for your thoughtful follow-up and constructive suggestion. After carefully reconsidering your comment, we realized that the original Figure 1 was intended as an illustrative example rather than a specific task instance from our main experiments. Following your suggestion, we have now grounded this motivating example in the concrete tiny-4ag-hard task and supplemented it with quantitative evidence to better demonstrate our motivation and contribution.

Specifically, we have made the following revisions:

Revised Figure 1 and its caption. We have updated Figure 1 to include the estimated information gap and absolute performance for MAGPO and MAIGPO. We have also revised the caption to explicitly identify the experimental setting: "Motivating example on tiny-4ag-hard."

Added the corresponding discussion in the Introduction (Lines 69–79). We have added a description of the experimental results.

Your suggestion has helped us make the motivation and central contribution of this work clearer and better supported by empirical evidence. We sincerely appreciate your time and constructive feedback.

Reviewer_sNSvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the so-called, Multi-Agent Information-Guided Policy Optimization (MAIGPO), for which it builds upon an existing work of Multi-Agent Guided Policy Optimization (MAGPO). It uses a centralized guider during training and decentralized learners for execution. The guider conditions on global information, and each learner uses its local observations only. The paper therefore proposes a set of methodologies to tackle issues under this setting.

This paper has two main components, the first is the bidirectional information alignment that aligns the representation between guider and learners, as a way to exchange the information between them. The author uses infoNCE loss for here. Secondly, It uses a variational information bottleneck (VIB) to extract the decision-relevant representations for both the guider and learners.

The main evaluation covers 15 RobotWarehouse tasks and 8 OvercookedV2 tasks, with additional experiments on MPE and SMAX. Comparisons include CTDE, CTDS, and CTCE methods.

Critical strengths

The paper is well written and the motivation is clear, it reflects the real-world examples where information may be available at the centralized and decentralized levels, i.e., global information is available during centralized training and decentralized agents acts only on the local observations.

The papers has appropriately combined few well known techniques, including the variational information bottleneck (VIB) and InfoNCE techniques to align the information between the guider and learner.

The paper also provides a good ablation studies amongst the many components it used.

Critical weaknesses

While this paper is interesting, it however, is still an incremental improvement from the previous work, namely an incremental extension of MAGPO. The authors have combined a few well known techniques, including InfoNCE and VIB together to improve the MAGPO.

Although the ablation studies has shown the advantage of each individual component, it's not directly establish that a combination of InfoNCE and VIB will help to achieve the optimal performance. the author also acknowledge that since the optimal action is unknown, the decision-relevant information gap cannot be estimated directly.

In addition, a minor suggestion is that, although I understand the setting of this paper may be different. Bud I would like the author to consider making some reference from other well known relevant teaching-student framework that is abundant in ML literatures. For example, one of the most famous paper that has a resemblance is the self-distillation with no labels (DINO), where it also has the teacher and student model, and the teacher sees the global image features and the student sees only the local ones. They roughly correspond to your guider and learner.

Questions for authors

n/a

Reviewer_79reOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed MAIGPO, which extends MAGPO, under centralized training with decentralized execution (CTDE). A guider uses global information during training, while deployed agents act using only local observations. Beyond MAGPO’s action matching, it adopts bidirectional contrastive learning to align guider and learner representations. Learners capture useful guider patterns, while the guider adapts toward learner-accessible information. Experiments across 23 tasks report performance improvements, although the method cannot recover information genuinely absent from local inputs.

Critical strengths
  • Useful extension to teacher-guided learning. MAIGPO adds information compression and bidirectional feature alignment to MAGPO’s action matching, helping centralized guidance better suit decentralized learners.

  • Broad experimental evaluation. Results across 23 tasks, together with component ablations, provide evidence of improved performance over MAGPO and other baselines.

Critical weaknesses
  1. The novelty over MAGPO needs clearer justification. The main additions use established information-bottleneck and contrastive-learning techniques. The paper should explain more clearly what new methodological insight their combination provides.

  2. The theory does not sufficiently justify the practical method. The analysis assumes that representations preserve all and only decision-relevant information. The training objectives encourage compression and alignment, but do not establish that this assumption holds.

  3. The information-gap explanation lacks direct evidence. The guider–learner return difference is used as a proxy for the information gap. However, similar returns do not necessarily indicate similar information, and aligned features may include details unrelated to decisions.

Questions for authors

When useful global information cannot be inferred from local observations, how does the method balance retaining that information in the guider against making its representation easier for learners to match?

Comments on the paper

1 comment

Dear Reviewers,

The authors have provided rebuttals to your review. Could you please check their responses and provide any further feedback or questions for the authors to address at your convenience?

Thank you for your time and contribution.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.