- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies learning-aware MARL in social dilemmas. All agents are RL agents trained with PPO and GRU policies, and no language model is involved.The paper makes three contributions. First, it proposes a learning-as-action framework, which treats each agent's learning process in an inner POSG as an action in an outer POSG, and treats information about co-players' learning as outer observations. Second, it defines a learning awareness order (LA-order) through set inclusion between the "types" of information each agent has about the others' learning. Third, it proposes LARL. An LA$k$ agent treats consecutive inner episodes as one outer episode and keeps its GRU hidden state across them, in the style of RL². Because is an integer multiple of , its trajectory spans several co-player updates. Training first pairs LA$k$ with LA$(k-1)$ agents and then shifts to same-order pairs using the M-FOS scheduler. A population-based variant is also given.
Critical strengths
The problem is relevant, since exploiting co-players' learning dynamics in social dilemmas is a central question in cooperative AI. LARL is simple, model-free and decentralized, and it needs no access to co-player gradients or value functions, only its own experience across co-player updates. Table 1 is a useful comparison of the information and assumptions behind existing methods. On Coins and Coop-Mining, LARL's cooperation scores are clearly higher than the baselines (Table 6), and the behavioral analyses in Figures 9 and 10 show that it does learn more cooperative strategies.
Critical weaknesses
W1. The LA-order is not well defined, and under its own definition LARL's different orders would be ranked equal. The order depends on a labeling map into a shared set of information types . The paper never specifies this set or how information is divided into types, and only assumes that "each listed type is relevant". Under this definition, LARL's LA2 and LA3 agents both receive one type, interaction histories across learning steps (Table 1), and differ only in the number of episodes they cover. By Eq. (2) the two sets contain each other, so LA^2 and LA^3 should share an order. The paper ranks them differently, which means the ranking comes from the ratio of update frequencies rather than from the definition. In addition, the abstract claims that an LA$^(k+1)$ agent can exploit an LA$^k$ agent, but no theorem or formal argument supports this.
W2. LARL seems to be PPO with a GRU whose hidden state persists across inner episodes, combined with the M-FOS grouping scheduler. The differences from M-FOS and COALA listed in Appendix A.1 are implementation choices. No ablation shows which of these drives the gains on Coins and Coop-Mining. Ablations may include removing hidden-state persistence across episodes or varying . The most direct baselines are also missing, the aurthor did not test independent PPO with a GRU in self-play and an ungrouped RL² agent.
W3. The paper claims superior ad-hoc generalization, but M-FOS and COALA are stronger in Coop-Mining (Figure 18), and the population variant LARL_2 is the worst agent in Coins (Figure 17, average ). In Cleanup (Figure 7), the gaps of 0.01 to 0.05 are far below the standard deviation of about 0.3.
Questions for authors
Q1. Under Eq. (2), do LARL's LA2 and LA3 agents have the same information types? If so, why are they not the same order? How are and defined?
Q2. What happens without hidden-state persistence across episodes, with different , or without the scheduler? How do independent PPO with a GRU and an ungrouped RL² agent perform?
Q3. How do the authors reconcile Figures 17 and 18 with the claim of superior ad-hoc generalization? Did all methods receive the same tuning budget?
6 comments
Response to Reviewer jVY8 (2/4)
(continuing)
Q1 & W1.
Q1 About LA formulation. Under Eq. (2), do LARL's LA2 and LA3 agents have the same information types? If so, why are they not the same order? How are and defined?
We thank the reviewers for identifying an insufficient discussion between the LA-order framework and LARL. To address this, we revised Section 3.3 to clarify the role and scope of this map, Section 4 to explicitly derive LARL's pairwise information types and Table 7 on how LARL corresponds to Section 3, and Table 1 to show type labels.
Justification of type function. In Section 3.3, and are specified by algorithm designers to compare heterogeneous learners' information, without assuming shared architectures or numerical information measures. LA-orders are relative to this abstraction. Type labels provide a unified lens for understanding learning awareness across algorithms; they are not algorithm inputs.
LARL's instantiation. In LARL, we aim to minimize the assumption for generality: each agent sees only its own trajectory. This trajectory can contain information about a coplayer 's learning through the behavioral consequences across 's updates. We provided an overview of notation of LARL related to Section 3 in our new Table 7 in the appendix, and summarize to minimize confusion as follows.
Let denote agent 's policy-parameter update interval, measured in inner episodes. We use to label interaction experience spanning updates of a coplayer with interval . Here, denotes information about 's learning available to through its own history. It contributes the type to only if it includes experience generated both before and after at least one update of coplayer . Otherwise, it contributes no learning-awareness type. Example 1 in the appendix shows an instance. In this way, the distribution shift from coplayer's learning (and subsequent new behavior) can be captured by the trajectory. These labels describe the training configuration and available experience; agents need not know the coplayers' update intervals or receive update indicators.
Under LARL's aligned, nested update schedule, a less-frequent-updating agent's window spans updates of a more-frequent-updating coplayer, but not vice versa. As an example, consider pairwise interactions among three agents with aligned intervals , where and are integers greater than one (Section 4 line 258). Then,
Equation (2) therefore gives the strict ordering , and Definition 3.2 assigns orders , , and , respectively, within this subset. The ordering follows from asymmetric access to experience spanning coplayer updates, rather than from treating a longer history as intrinsically more informative.
A simple example shows that in LARL, less-frequent-updating agent can observe how a more-frequent-updating coplayer learns through the consequences in its own experience: in IPD, both agents start with always-C. As the coplayer learns to defect, the ego agent can infer this change from its own observations and rewards.
Our examples establish LA-order using simple information types, but the framework also supports multiple types in . See our response to Q3 of Reviewer 1j9Z for future directions.
About exploitation guarantee. Our construction characterizes the information available and used under the specified training protocol, without implying a general theoretical exploitation guarantee. We qualify the abstract's exploitation statement as an empirical finding in the evaluated settings. Section 5.3 presents experiments on exploitation configurations, with Figure 8 illustrating the mean reward curves and Table 3 summarizing the exploitation gap at the last training iteration. Empirical evidences suggest learning-aware method's exploitation behavior via their stronger -- at least not weaker -- overall performance. Asymmetric learning information does not guarantee exploitation in general; instead the learning outcome is quite domain-specific. In a pure cooperative task, for instance, an exploitative strategy may not exist. As discussed in Appendix B.3, Stag Hunt domains may not pose a significant social learning challenge, as shown by the small exploitation gaps in Coop Mining and ISH.
(continued next)
Response to Reviewer jVY8 (3/4)
(continuing)
Q2 & W2.
Remarks on LARL's novelty and contribution
Q2.1 Remarks on LARL's novelty and contribution. LARL seems to be PPO with a GRU whose hidden state persists across inner episodes, combined with the M-FOS grouping scheduler.
To interpret LARL's novelty, we would like to clarify that our method is not characterized solely by its architecture and policy optimization approach; its data collection and training schemes constitute critical components of its learning mechanism. In a multi-agent system, agents interact and learn in a common environment, leading to inter-dependent decision-making and learning processes. Even the same agent, uniletarally following the same rollout and optimization strategy, could result in different learning outcomes if the coplayers are different. We establish an formulation of learning awareness and LA-orders, upon which we propose LARL to further our understanding of learning dynamics. We observe important findings not seen in prior work. To elaborate, in Sec 5.4 Figure 6, the identially configured LA2 agent exhibits completely different behaviors when trained with different coplayers -- LA1 in (a,e) or LA3 in (c,g). On the other hand, (a,e), (b,f), (c,g) show similar behavior despite their distinct groupings, demonstrating LA-order's relative nature. Additionally, LARL enables multi-LA-order support, and presents LA3-vs-LA2-vs-LA1 and their interesting chain reaction in (d,h). These new findings justify the critical role of training schemes in MARL, and highlight the importance of the LA-order and LARL in studying multi-agent learning dynamics.
Therefore, LARL's algorithmic differences from M-FOS/COALA are beyond merely implementation choices. Prior methods only support LA2 learning, which prevent them from showing the relevant findings in Sec 5.4 & Sec F.4 to establish the same understanding of learning-awareness. Ablation experiments below also demonstrate that implementation choices like memory-persistent RNN or the linear grouping scheduler are not necessary conditions for effective learning.
Q2.2 Ablate network architecture. What happens without hidden-state persistence across episodes ...
We thank the reviewer for pointing out this potential source of confusion. First, we clarify the role of the RNN/GRU. The primary reason we adopt an RNN is to address the ubiquitous partial observability (PO) challenge in Deep RL [1,2], as we lift the full observability assumption for generality. The RNN is not an integral component of LARL but an implementation choice. Using the same architecture across all experiments ensures fair comparisons and minimizes tuning. In the updated revision, we moved related descriptions from Section 4 to Appendix I.2 to avoid potential confusion.
Then, to isolate the effect of cross-episode memory, we compare the RNN-based LARL to its MLP variant which completely eliminates the RNN's effect, in two nearly fully observable tasks, CleanUp and Coins (with only zapped_timer and step_count unobservable), and all matrix games.
Figure 20,21,22 show that LARL-MLP performs similarly to LARL-RNN across tasks, quantitatively and qualitatively, isolating network architectures' effect for effective learning.
Q2.3 Experiments on constant coplayer sampling schedule. ... without the scheduler?
We adopt the linear sampling schedule [3] for all experience-based methods to promote a fair comparison and minimal hyperparameter tuning. In Sec H.2, we present ablations that apply a constant coplayer sampling schedule, sweeping probabilities , where and denote the pure exploitation setting and LA2-intra-play, respectively, and intermediate values sample both LA-order coplayers throughout training. Figure 23,24 illustrate constant-scheduled LARL performs comparably to linear-scheduled LARL from Figure 4, given properly tuned . Such empirical evidence strengthens the importance of training an agent with both same and adjusted LA-order coplayers.
Q2.4 Intra-play baselines. How do independent PPO with a GRU and an ungrouped RL$^2$ agent perform?
For consistency, we use "intra-play" to denote self-play without parameter sharing, because whether shared parameters and gradient updates violate the decentralized learning assumption remains controversial.
We present related results in Sec H.3. Figure 25 shows that intra-play, with either LA1/LA2 config, downperforms LARL in Coins, Prisoner's Dilemma, and Coop Mining, especially in Prisoner's Dilemma where it fails completely. In Figure 26,27, LA2-intra-play perform more similarly to LA1-intra-play than LARL, which endorses LA-order's relativeness and mutual influences on learning dynamics over adjacent LA-orders. Figure 31 and Table 8 also show that strong training-time performance in CleanUp does not guarantee strong ad-hoc cross-play performance.
(continued next)
Response to Reviewer jVY8 (1/4)
We sincerely appreciate your constructive feedback, which has helped us reflect on where our presentation may lacked clarity and how to better communicate our intended ideas.
In the updated revision, we marked updates in the main text in blue, and included new experiments and implementation details in Appendix G and Appendix I, respectively. We list related experiments in the following table for easy reference.
*To avoid interference with figure/table numbering, we kept the original figures/tables in previous order and appended new ones to the end of the appendix.
Please kindly find our detailed responses below.
Q3 & W3.
In Figure 17, the worst average −0.370 corresponds to LARL_2's coplayers in average, rather than LARL_2 itself, the best agent in Coins (+0.368).
The intended interpretation of cross-play result is: In each cell, the upper and lower triangles correspond to the column and row, respectively, as illustrated in the following table.
& & \text{Agent A} \\\\ \hline & & \text{Perf. A} \\\\ \smash{\text{Agent B}} & \text{Perf. B} & \\\\ \hline \end{array}$$ > the gaps of 0.01 to 0.05 are far below the standard deviation of about 0.3. The large variances are not only due to evaluation with mutually unseen agents, but also to the inherent reward sparsity from the tasks. A trajectory with both 0's and positive rewards could result in large variance in the empirical mean. To this end, for the *mean reward* Figures 7,17,18, in section F.5, we correspondingly provide *mean returns* results in Figure 28,29,30, and additional results in Figure 31,32,33 with large agent pools, with per-method statistics summarized in Table 8,9,10, respectively. Additionally, we do not claim LARL is the strongest in every task, but it is the strongest in terms of overall performance, while all experience-based methods perform comparably in Coop Mining. Although M-FOS and COALA are stronger in Coop-Mining, LARL outperforms both in CleanUp and Coins. > **Q3.2 Remarks on hyperparameters.** Did all methods receive the same tuning budget? Sec I.1 provides the complete tuning procedure and hyperparameter values (Table 14-21), which specify the default settings across methods & tasks, and per-method overrides. LARL, M-FOS, and COALA receive the same tuning budget, while the gradient-based LOLA has one fewer hyperparameter to tune. *(continued next)*