- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MAIGPO, which views the guider-learner imitation gap in CTDS as an information-theoretic mismatch between the decision-relevant information available to the centralized guider and the decentralized learner. Under an idealized assumption, it shows that reducing this gap amounts to maximizing the mutual information between their latent representations. To achieve this, MAIGPO augments MAGPO’s action-level imitation objective with a variational information bottleneck and a bidirectional contrastive alignment loss based on InfoNCE. Experiments across 23 tasks in RobotWarehouse and OvercookedV2 show that MAIGPO outperforms CTDE, CTDS, and CTCE baselines.
Critical strengths
-
Principled derivation of the method. The paper derives that closing the information gap reduces to maximizing the shared information between guider and learner representations, giving it formal justification.
-
Empirical validation. 23 tasks across two benchmark suites, comparisons against CTDE/CTCE/CTDS baselines, proper statistical reporting (IQM, bootstrapped CIs, probability-of-improvement), plus additional results on MPE/SMAX.
-
Ablations. The authors isolate the contribution of the information bottleneck vs. the alignment loss, and specifically tests whether gains grow with the information gap itself, not just reporting aggregate performance.
Critical weaknesses
-
The motivating example lacks detail and clarity. Figure 1 illustrates MAIGPO’s architecture rather than the failure it aims to address. Given that the information-gap interpretation is one of the paper’s central novelties and pivotal to the rest of the work, the authors should provide a concrete example or evidence showing that this gap exists and explaining why MAGPO’s action-level alignment fails to address it.
-
Unclear and potentially insufficient seed count. The paper states each algorithm is trained with “1–5 seeds per task” but doesn’t specify how many seeds were used for which tasks. Results based on 1–2 seeds are not statistically meaningful [1]; they can’t support the confidence intervals reported (Section 5.1) and make it impossible to assess whether reported differences between MAIGPO and baselines reflect genuine improvement or seed variance, particularly on the 23 tasks evaluated. This ambiguity should be resolved with a per-task seed count, and any task run with fewer than ~3 seeds should be flagged or excluded from headline comparisons.
-
Unverified link between task difficulty and the information gap. Section 5.2 orders tasks by an assumed increasing information gap (based on map size/agent count/queue size) and shows MAIGPO’s advantage over MAGPO grows accordingly, yet the gap itself is never directly measured, only inferred from task structure. This is consistent with the paper’s hypothesis but doesn’t confirm it, since harder tasks could favor MAIGPO for reasons unrelated to the proposed mechanism.
I would be happy to raise my score if the authors can adequately address those issues.
[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
Questions for authors
-
How many seeds were used per task specifically, and were any of the headline results (Figures 3–4) based on just 1–2 seeds?
-
Given that , , , , and are tuned separately per task (Appendix F.1), how sensitive is MAIGPO's advantage over MAGPO to this tuning budget — would the gains hold with a single shared configuration across tasks, or only under per-task search?
7 comments
Dear authors, Thank you for the detailed response and for the effort you put into addressing my concerns. I appreciate the new results and the clarification regarding the seed count.
One concern remains. The motivating example in the introduction (Fig. 1) sets the stage for the entire paper, so it should be supported by evidence that the gap exists, in terms of either the guide–learner return gap or the information gap. Could the new estimator you proposed be applied to this example, for instance, by measuring the estimated gap at states matching the Fig. 1 scenario for MAGPO and MAIGPO?
Since the information-gap perspective is the paper’s central novelty and the main motivation for MAIGPO, the example illustrating it should be supported by evidence. The new evaluation results (Fig. 5) do this well; the motivating example in the introduction deserves similar support.
Dear Reviewer,
Thank you very much for your thoughtful follow-up and constructive suggestion. After carefully reconsidering your comment, we realized that the original Figure 1 was intended as an illustrative example rather than a specific task instance from our main experiments. Following your suggestion, we have now grounded this motivating example in the concrete tiny-4ag-hard task and supplemented it with quantitative evidence to better demonstrate our motivation and contribution.
Specifically, we have made the following revisions:
Revised Figure 1 and its caption. We have updated Figure 1 to include the estimated information gap and absolute performance for MAGPO and MAIGPO. We have also revised the caption to explicitly identify the experimental setting: "Motivating example on tiny-4ag-hard."
Added the corresponding discussion in the Introduction (Lines 69–79). We have added a description of the experimental results.
Your suggestion has helped us make the motivation and central contribution of this work clearer and better supported by empirical evidence. We sincerely appreciate your time and constructive feedback.
Thank you for your positive assessment and constructive feedback
We appreciate the reviewers' careful evaluation and valuable feedback. The comments have helped us refine our arguments and better communicate the contributions of our work. We respond to each question and concern below.
Figure 1 illustrates the motivation behind MAIGPO: the guider–learner imitation gap reflects a mismatch in decision-relevant information. For example, consider the first agent in the highlighted box on the right of Fig. 1: the guider selects the upper item based on global information, allowing the other agent to collect the item nearest to it, whereas the learner selects the lower item because of its limited observations.
MAGPO’s policy alignment matches the final action distributions, influencing hidden representations only indirectly. We do not claim that action-level alignment fails to address this mismatch, but that it may be insufficient. We therefore introduce information alignment to explicitly promote decision-relevant information sharing between the guider and learner, complementing the existing policy-level constraints. Extensive experiments further demonstrate the performance gains achieved by our method.
Additionally, we use the guider–learner return difference (Figure 6) as a proxy for the information gap I(Ztg;At)−I(Ztl;At). The figure shows that MAIGPO generally achieves a smaller return gap and higher performance than MAGPO, particularly on forced_coord. These results are consistent with our motivation and theoretical perspective.
We apologize for the ambiguity. All algorithms on all tasks were trained with five independent random seeds: 1, 2, 3, 4, and 5. “1–5 seeds” was intended to denote these seed values, not a varying seed count. We will correct this wording in Section 5.1. Additionally, as described in Appendix E.1, we set os.environ["XLA_FLAGS"] = "--xla_gpu_deterministic_ops=true" in all experiments to enable deterministic GPU operations and improve reproducibility.
Our ordering reflects structural differences between guider and learner inputs, rather than task difficulty alone. With a fixed local sensing range, larger maps and more agents can increase the potentially relevant information unavailable to individual learners. Smaller request queues require tighter task allocation to avoid competing for the same target; the guider can coordinate these assignments using the global state and preceding agents’ actions, whereas learners lack direct access to this information.
These factors motivate the ordering as an approximate proxy for information asymmetry. The increasing advantage of MAIGPO over MAGPO further suggests that MAIGPO is better suited to settings with greater information asymmetry.
We interpret Fig. 8 as supporting evidence consistent with our hypothesis, rather than direct confirmation of the mechanism. We will revise the section and caption to clarify that the ordering serves as a structural proxy for information asymmetry.
All experiments across all tasks and algorithms use the same five random seeds (1, 2, 3, 4, and 5) to ensure fair comparisons.
Our hyperparameter tuning procedure largely follows MAGPO, which uses Optuna-TPE with 40 trials per task. Similarly, MAIGPO uses 10 trials to tune the IB-related hyperparameters, followed by 30 trials to tune the remaining hyperparameters. We also adopt MAGPO's reported optimal hyperparameters for the underlying algorithm. Our results demonstrate that MAIGPO outperforms MAGPO under task-specific tuning, suggesting that the improvements are not merely attributable to a different tuning procedure.
Given the differences in task characteristics, a single shared configuration may not fully preserve the gains achieved through task-specific tuning. Following MAGPO, we adopt task-specific optimized hyperparameters. We will release the full parameter configurations for all tasks with our open-source code.