acceptodds
Under review as a conference paper at ICLR 2027

Learning Awareness for Effective and Efficient Learning in Social Dilemmas

Abstract

Many real-world multi-agent systems involve social dilemmas, where agents have partially aligned but conflicting interests, and mutual cooperation can benefit agents in long run. Social dilemmas are challenging, as self-interested agents often choose strategies that are suboptimal, particularly when other agents are also learning. Recent learning-aware methods offer a promising direction, but often make restrictive assumptions about how agents understand the coplayers' learning processes. In this work, we conceptualize learning as action within a decision model to help characterize learning awareness (LA). We accordingly introduce the learning awareness order (LA-order) over learning agents, where an LA(k+1) agent can "exploit" an LAk coplayer by leveraging its information about the coplayer's learning process. The LA-order naturally provides a lens to understand a wide spectrum of learners, ranging from fixed random agents (LA0), self-interested naive learners LA1, to prior learning-aware agents (LA2). Based on this understanding, we introduce a model-free, decentralized multi-agent reinforcement learning (MARL) approach extensible to any LA-orders, which performs learning-aware RL through dynamic grouping of agents, with each agent leveraging solely its own experience. Through evaluation across diverse social dilemma challenges, in both iterated matrix games and state-based benchmarks, we demonstrate the effectiveness of our algorithms, including under ad-hoc generalization, and the value of the LA-order. Experiments also show that agents with adjacent LA-orders mutually influence each other's learning dynamics.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, multi-agent systems, social dilemma, ad-hoc generalization, learning awareness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jVY8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning-aware MARL in social dilemmas. All agents are RL agents trained with PPO and GRU policies, and no language model is involved.The paper makes three contributions. First, it proposes a learning-as-action framework, which treats each agent's learning process in an inner POSG as an action in an outer POSG, and treats information about co-players' learning as outer observations. Second, it defines a learning awareness order (LA-order) through set inclusion between the "types" of information each agent has about the others' learning. Third, it proposes LARL. An LA$k$ agent treats TkT_k consecutive inner episodes as one outer episode and keeps its GRU hidden state across them, in the style of RL². Because TkT_k is an integer multiple of Tk−1T_{k-1}, its trajectory spans several co-player updates. Training first pairs LA$k$ with LA$(k-1)$ agents and then shifts to same-order pairs using the M-FOS scheduler. A population-based variant is also given.

Critical strengths

The problem is relevant, since exploiting co-players' learning dynamics in social dilemmas is a central question in cooperative AI. LARL is simple, model-free and decentralized, and it needs no access to co-player gradients or value functions, only its own experience across co-player updates. Table 1 is a useful comparison of the information and assumptions behind existing methods. On Coins and Coop-Mining, LARL's cooperation scores are clearly higher than the baselines (Table 6), and the behavioral analyses in Figures 9 and 10 show that it does learn more cooperative strategies.

Critical weaknesses

W1. The LA-order is not well defined, and under its own definition LARL's different orders would be ranked equal. The order depends on a labeling map type(⋅)\mathrm{type}(\cdot) into a shared set of information types Y\mathcal{Y}. The paper never specifies this set or how information is divided into types, and only assumes that "each listed type is relevant". Under this definition, LARL's LA2 and LA3 agents both receive one type, interaction histories across learning steps (Table 1), and differ only in the number of episodes TkT_k they cover. By Eq. (2) the two sets contain each other, so LA^2 and LA^3 should share an order. The paper ranks them differently, which means the ranking comes from the ratio of update frequencies rather than from the definition. In addition, the abstract claims that an LA$^(k+1)$ agent can exploit an LA$^k$ agent, but no theorem or formal argument supports this.

W2. LARL seems to be PPO with a GRU whose hidden state persists across inner episodes, combined with the M-FOS grouping scheduler. The differences from M-FOS and COALA listed in Appendix A.1 are implementation choices. No ablation shows which of these drives the gains on Coins and Coop-Mining. Ablations may include removing hidden-state persistence across episodes or varying TkT_k. The most direct baselines are also missing, the aurthor did not test independent PPO with a GRU in self-play and an ungrouped RL² agent.

W3. The paper claims superior ad-hoc generalization, but M-FOS and COALA are stronger in Coop-Mining (Figure 18), and the population variant LARL_2 is the worst agent in Coins (Figure 17, average −0.370-0.370). In Cleanup (Figure 7), the gaps of 0.01 to 0.05 are far below the standard deviation of about 0.3.

Questions for authors

Q1. Under Eq. (2), do LARL's LA2 and LA3 agents have the same information types? If so, why are they not the same order? How are type(⋅)\mathrm{type}(\cdot) and Y\mathcal{Y} defined?

Q2. What happens without hidden-state persistence across episodes, with different TkT_k, or without the scheduler? How do independent PPO with a GRU and an ungrouped RL² agent perform?

Q3. How do the authors reconcile Figures 17 and 18 with the claim of superior ad-hoc generalization? Did all methods receive the same tuning budget?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jVY8 (1/4)

We sincerely appreciate your constructive feedback, which has helped us reflect on where our presentation may lacked clarity and how to better communicate our intended ideas.

In the updated revision, we marked updates in the main text in blue, and included new experiments and implementation details in Appendix G and Appendix I, respectively. We list related experiments in the following table for easy reference.

ExperimentsSectionsRemarks
Cross-play (new)Appendix F.5; Figure 28-33, Table 8-10additional results and clean summaries
distinct LA-order group combinationsSec 5.4 & F.4, Figure 6,12-14same agent trained with different LA-order coplayer shows different outcomes; training scheme plays a critical role in defining a method
MLP vs RNN in LARL (new)Appendix H.1; Figure 20,21,22separate the effect of RNN hidden state; note the comparable performance of MLP-variant, showing generality of LARL
constant grouping schedule (new)Appendix H.2; Figure 23,24fixed probability of sampling lower-LA-order coplayer; emphasize importance of grouping with both LA-order coplayers
LA1-intra-play, LA2-intra-play (new)Appendix H.3; Figure 25,26,27,31, Table 8equally-update-frequent agents trained together; ablate update interval and grouping

*To avoid interference with figure/table numbering, we kept the original figures/tables in previous order and appended new ones to the end of the appendix.

Please kindly find our detailed responses below.

Q3 & W3.

Q3.1 Clarification on cross-play results. How do the authors reconcile Figures 17 and 18 with the claim of superior ad-hoc generalization? ... M-FOS and COALA are stronger in Coop-Mining (Figure 18), and the population variant LARL_2 is the worst agent in Coins (Figure 17, average −0.370-0.370 ...

In Figure 17, the worst average −0.370-0.370 corresponds to LARL_2's coplayers in average, rather than LARL_2 itself, the best agent in Coins (+0.368+0.368).

The intended interpretation of cross-play result is: In each cell, the upper and lower triangles correspond to the column and row, respectively, as illustrated in the following table.

& & \text{Agent A} \\\\ \hline & & \text{Perf. A} \\\\ \smash{\text{Agent B}} & \text{Perf. B} & \\\\ \hline \end{array}$$ > the gaps of 0.01 to 0.05 are far below the standard deviation of about 0.3. The large variances are not only due to evaluation with mutually unseen agents, but also to the inherent reward sparsity from the tasks. A trajectory with both 0's and positive rewards could result in large variance in the empirical mean. To this end, for the *mean reward* Figures 7,17,18, in section F.5, we correspondingly provide *mean returns* results in Figure 28,29,30, and additional results in Figure 31,32,33 with large agent pools, with per-method statistics summarized in Table 8,9,10, respectively. Additionally, we do not claim LARL is the strongest in every task, but it is the strongest in terms of overall performance, while all experience-based methods perform comparably in Coop Mining. Although M-FOS and COALA are stronger in Coop-Mining, LARL outperforms both in CleanUp and Coins. > **Q3.2 Remarks on hyperparameters.** Did all methods receive the same tuning budget? Sec I.1 provides the complete tuning procedure and hyperparameter values (Table 14-21), which specify the default settings across methods & tasks, and per-method overrides. LARL, M-FOS, and COALA receive the same tuning budget, while the gradient-based LOLA has one fewer hyperparameter to tune. *(continued next)*
AuthorsOpenReview ↗

Response to Reviewer jVY8 (2/4)

(continuing)

Q1 & W1.

Q1 About LA formulation. Under Eq. (2), do LARL's LA2 and LA3 agents have the same information types? If so, why are they not the same order? How are Y\mathcal{Y} and type⁡(⋅)\operatorname{type}(\cdot) defined?

We thank the reviewers for identifying an insufficient discussion between the LA-order framework and LARL. To address this, we revised Section 3.3 to clarify the role and scope of this map, Section 4 to explicitly derive LARL's pairwise information types and Table 7 on how LARL corresponds to Section 3, and Table 1 to show type labels.

Justification of type function. In Section 3.3, Y\mathcal{Y} and type⁡\operatorname{type} are specified by algorithm designers to compare heterogeneous learners' information, without assuming shared architectures or numerical information measures. LA-orders are relative to this abstraction. Type labels provide a unified lens for understanding learning awareness across algorithms; they are not algorithm inputs.

LARL's instantiation. In LARL, we aim to minimize the assumption for generality: each agent ii sees only its own trajectory. This trajectory can contain information about a coplayer jj's learning through the behavioral consequences across jj's updates. We provided an overview of notation of LARL related to Section 3 in our new Table 7 in the appendix, and summarize to minimize confusion as follows.

Let TiT_i denote agent ii's policy-parameter update interval, measured in inner episodes. We use HT\mathsf{H}_T to label interaction experience spanning updates of a coplayer with interval TT. Here, y~i,j\tilde{y}^{i,j} denotes information about jj's learning available to ii through its own history. It contributes the type HTj\mathsf{H}_{T_j} to yijy_{ij} only if it includes experience generated both before and after at least one update of coplayer jj. Otherwise, it contributes no learning-awareness type. Example 1 in the appendix shows an instance. In this way, the distribution shift from coplayer's learning (and subsequent new behavior) can be captured by the trajectory. These labels describe the training configuration and available experience; agents need not know the coplayers' update intervals or receive update indicators.

Under LARL's aligned, nested update schedule, a less-frequent-updating agent's window spans updates of a more-frequent-updating coplayer, but not vice versa. As an example, consider pairwise interactions among three agents with aligned intervals T1<T2<T3T_1 < T_2 < T_3, where T2/T1T_2/T_1 and T3/T2T_3/T_2 are integers greater than one (Section 4 line 258). Then, y23=∅⊊y32=HT2,y_{23} = \varnothing \subsetneq y_{32} = \\{\mathsf{H}_{T_2}\\},

y13=∅⊊y31=HT1,y_{13} = \varnothing \subsetneq y_{31} = \\{\mathsf{H}_{T_1}\\},

y12=∅⊊y21=HT1.y_{12} = \varnothing \subsetneq y_{21} = \\{\mathsf{H}_{T_1}\\}. Equation (2) therefore gives the strict ordering 3≻2≻13 \succ 2 \succ 1, and Definition 3.2 assigns orders 33, 22, and 11, respectively, within this subset. The ordering follows from asymmetric access to experience spanning coplayer updates, rather than from treating a longer history as intrinsically more informative.

A simple example shows that in LARL, less-frequent-updating agent can observe how a more-frequent-updating coplayer learns through the consequences in its own experience: in IPD, both agents start with always-C. As the coplayer learns to defect, the ego agent can infer this change from its own observations and rewards.

Our examples establish LA-order using simple information types, but the framework also supports multiple types in yi,jy_{i,j}. See our response to Q3 of Reviewer 1j9Z for future directions.

About exploitation guarantee. Our construction characterizes the information available and used under the specified training protocol, without implying a general theoretical exploitation guarantee. We qualify the abstract's exploitation statement as an empirical finding in the evaluated settings. Section 5.3 presents experiments on exploitation configurations, with Figure 8 illustrating the mean reward curves and Table 3 summarizing the exploitation gap at the last training iteration. Empirical evidences suggest learning-aware method's exploitation behavior via their stronger -- at least not weaker -- overall performance. Asymmetric learning information does not guarantee exploitation in general; instead the learning outcome is quite domain-specific. In a pure cooperative task, for instance, an exploitative strategy may not exist. As discussed in Appendix B.3, Stag Hunt domains may not pose a significant social learning challenge, as shown by the small exploitation gaps in Coop Mining and ISH.

(continued next)

AuthorsOpenReview ↗

Response to Reviewer jVY8 (3/4)

(continuing)

Q2 & W2.

Remarks on LARL's novelty and contribution

Q2.1 Remarks on LARL's novelty and contribution. LARL seems to be PPO with a GRU whose hidden state persists across inner episodes, combined with the M-FOS grouping scheduler.

To interpret LARL's novelty, we would like to clarify that our method is not characterized solely by its architecture and policy optimization approach; its data collection and training schemes constitute critical components of its learning mechanism. In a multi-agent system, agents interact and learn in a common environment, leading to inter-dependent decision-making and learning processes. Even the same agent, uniletarally following the same rollout and optimization strategy, could result in different learning outcomes if the coplayers are different. We establish an formulation of learning awareness and LA-orders, upon which we propose LARL to further our understanding of learning dynamics. We observe important findings not seen in prior work. To elaborate, in Sec 5.4 Figure 6, the identially configured LA2 agent exhibits completely different behaviors when trained with different coplayers -- LA1 in (a,e) or LA3 in (c,g). On the other hand, (a,e), (b,f), (c,g) show similar behavior despite their distinct groupings, demonstrating LA-order's relative nature. Additionally, LARL enables multi-LA-order support, and presents LA3-vs-LA2-vs-LA1 and their interesting chain reaction in (d,h). These new findings justify the critical role of training schemes in MARL, and highlight the importance of the LA-order and LARL in studying multi-agent learning dynamics.

Therefore, LARL's algorithmic differences from M-FOS/COALA are beyond merely implementation choices. Prior methods only support LA2 learning, which prevent them from showing the relevant findings in Sec 5.4 & Sec F.4 to establish the same understanding of learning-awareness. Ablation experiments below also demonstrate that implementation choices like memory-persistent RNN or the linear grouping scheduler are not necessary conditions for effective learning.

Q2.2 Ablate network architecture. What happens without hidden-state persistence across episodes ...

We thank the reviewer for pointing out this potential source of confusion. First, we clarify the role of the RNN/GRU. The primary reason we adopt an RNN is to address the ubiquitous partial observability (PO) challenge in Deep RL [1,2], as we lift the full observability assumption for generality. The RNN is not an integral component of LARL but an implementation choice. Using the same architecture across all experiments ensures fair comparisons and minimizes tuning. In the updated revision, we moved related descriptions from Section 4 to Appendix I.2 to avoid potential confusion.

Then, to isolate the effect of cross-episode memory, we compare the RNN-based LARL to its MLP variant which completely eliminates the RNN's effect, in two nearly fully observable tasks, CleanUp and Coins (with only zapped_timer and step_count unobservable), and all matrix games. Figure 20,21,22 show that LARL-MLP performs similarly to LARL-RNN across tasks, quantitatively and qualitatively, isolating network architectures' effect for effective learning.

Q2.3 Experiments on constant coplayer sampling schedule. ... without the scheduler?

We adopt the linear sampling schedule [3] for all experience-based methods to promote a fair comparison and minimal hyperparameter tuning. In Sec H.2, we present ablations that apply a constant coplayer sampling schedule, sweeping probabilities λ∈{1.0,0.75,0.5,0.25,0.0}\lambda \in \{1.0, 0.75, 0.5, 0.25, 0.0\}, where λ=1.0\lambda=1.0 and λ=0.0\lambda=0.0 denote the pure exploitation setting and LA2-intra-play, respectively, and intermediate values sample both LA-order coplayers throughout training. Figure 23,24 illustrate constant-scheduled LARL performs comparably to linear-scheduled LARL from Figure 4, given properly tuned λ\lambda. Such empirical evidence strengthens the importance of training an agent with both same and adjusted LA-order coplayers.

Q2.4 Intra-play baselines. How do independent PPO with a GRU and an ungrouped RL$^2$ agent perform?

For consistency, we use "intra-play" to denote self-play without parameter sharing, because whether shared parameters and gradient updates violate the decentralized learning assumption remains controversial.

We present related results in Sec H.3. Figure 25 shows that intra-play, with either LA1/LA2 config, downperforms LARL in Coins, Prisoner's Dilemma, and Coop Mining, especially in Prisoner's Dilemma where it fails completely. In Figure 26,27, LA2-intra-play perform more similarly to LA1-intra-play than LARL, which endorses LA-order's relativeness and mutual influences on learning dynamics over adjacent LA-orders. Figure 31 and Table 8 also show that strong training-time performance in CleanUp does not guarantee strong ad-hoc cross-play performance.

(continued next)

Reviewer_1j9ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a decision model that extends Learning Awareness (LA) in multi-agent RL to tackle social dilemmas. The proposed model allows for an ordering of LA between agents; agents are ordered based on the types of information that they use about other agents’ learning processes. Consequently, higher LA agents could exploit lower LA agents. The authors then propose an algorithm, LARL, building on their model, and show in an empirical setting gains over other LA methods across social-dilemma environments.

Critical strengths
  1. The paper presents an interesting novel scheme in Learning Awareness. Namely, they establish a compact formal ordering scheme for Learning Awareness in agents.
  2. The results show a high reported mean return across environments, including particularly measurable improvements on Coins and Coop Mining.
Critical weaknesses
  1. The paper leaves substantial uncertainty in the design of LARL. The correspondence between theoretical definitions of LA and LA order in section 3.3 and the algorithm in section 4 is left implicit. Explicitly instantiating the learning-as-actions framework to the proposed LARL would greatly improve clarity. Essential explanations about the GRU memory and parameter updates are distributed throughout the Method, algorithm pseudocode, and appendices.
  2. The paper is missing important analysis of the algorithm. The experiments do not isolate the contribution of choices such as sequential group updates and the grouping schedule. The return estimation for policy updates is not explicitly specified; it is not clear to me whether it differs from standard per-trajectory methods.

While the paper introduces an interesting new framework to rank LA agents, I have concerns about the clarity and algorithmic choices in the paper.

Questions for authors
  1. On constructing the algorithm, could the authors specify the yijy_{ij} for each LA order of agents, and how they satisfy the ordering in equation 2? This is especially ambiguous given that recurrent-state adaptation, which lower LA agents could have access to, could count as learning (Appendix A.2). Also, I would encourage the authors to report final TkT_k values in each experiment.
  2. The authors contrast aggregated group updates in COALA and suggest sequential updates. Could the authors run their method LARL with gradients aggregated across groups to assess the contribution of the sequential updates? Similarly, could a fixed probability of sampling agents achieve similar performance instead of the suggested grouping scheme? Additionally, maybe the effect of cross-episode memory could be isolated by resetting memory for higher LA agents at every inner episode boundary.
  3. I think a promising future line of work is algorithmically applying the LA ranking to different types of information, similar to the discussions in Appendix A.2.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1j9Z (1/2)

We appreciate your constructive comments. Your valuable suggestions have significantly elevated our work.

In the updated revision, we marked updates in the main text in blue, and included new experiments and implementation details in Appendix G and Appendix I, respectively. We list related experiments in the following table for easy reference.

ExperimentsSectionsRemarks
MLP vs RNN in LARL (new)Appendix H.1; Figure 20,21,22separate the effect of RNN hidden state; note the comparable performance of MLP-variant, showing generality of LARL
constant grouping schedule (new)Appendix H.2; Figure 23,24fixed probability of sampling lower-LA-order coplayer; emphasize importance of grouping with both LA-order coplayers
distinct LA-order group combinationsSec 5.4 & F.4, Figure 6,12-14same agent trained with different LA-order coplayer shows different outcomes; training scheme plays a critical role in defining a method

*To avoid interference with figure/table numbering, we kept the original figures/tables in previous order and appended new ones to the end of the appendix.

Please kindly find our detailed responses below.

W1 & Q1.

Q1.1 LA formulation. Explicitly instantiating the learning-as-actions framework to the proposed LARL would greatly improve clarity.

We thank the reviewer to point it out. Now, Section 4 is updated (line 258-262) to reflect the connection on how LARL instantiate Section 3.3, and specified yijy_{ij} that also connects to equation 2. Please also refer to our response to Reviewer jVY8 (Q1 and W1) for a detailed explanation.

Q1.2 Clarification on network architecture. Essential explanations about the GRU memory ...

We would like to clarify different reasons for mentioning RNN in different sections. In Appendix A.2 (the current Appendix B), Example 2 provides an instantiation of outer state and its different intra-agent and inter-agent components. The RNN example intends to show that our notion of learning also encompasses in-context learning, beyond gradient-update learning. Note that agent 2's hidden state is accessible only to agent 2 itself.

In Sec 4 Method, we adopt an RNN to tackle the ubiquitous partial observability (PO) challenge in Deep RL [1,2], but it is rather an implementation choice. In the updated revision, we moved related descriptions to Appendix I.2 to avoid potential confusion. We also provide ablation experiments on network architecture discussed below.

Q1.3 Remarks on hyperparameters. I would encourage the authors to report final TkT_k values in each experiment.

Hyperparameters are summarized in Sec I.1 (Table 14-21), which specify the default settings across methods & tasks, and per-method overrides, including configs such as TkT_k and return estimation.

W2 & Q2.

Q2.1 Ablate network architecture. ... maybe the effect of cross-episode memory could be isolated ...

We thank the reviewer for pointing out this potential source of confusion. First, we clarify the role of the RNN/GRU. The primary reason we adopt an RNN is to address the ubiquitous partial observability (PO) challenge in Deep RL [1,2], as we lift the full observability assumption for generality. The RNN is not an integral component of LARL but an implementation choice. Using the same architecture across all experiments ensures fair comparisons and minimizes tuning. In the updated revision, we moved related descriptions from Section 4 to Appendix I.2 to avoid potential confusion.

Then, to isolate the effect of cross-episode memory, we compare the RNN-based LARL to its MLP variant which completely eliminates the RNN's effect, in two nearly fully observable tasks, CleanUp and Coins (with only zapped_timer and step_count unobservable), and all matrix games. Figure 20,21,22 show that LARL-MLP performs similarly to LARL-RNN across tasks, quantitatively and qualitatively, isolating network architectures' effect for effective learning.

Q2.2 Experiments on constant coplayer sampling schedule. could a fixed probability of sampling agents achieve similar performance instead of the suggested grouping scheme?

We adopt the linear sampling schedule [3] for all experience-based methods to promote a fair comparison and minimal hyperparameter tuning. In Sec H.2, we present ablations that apply a constant coplayer sampling schedule, sweeping probabilities λ∈{1.0,0.75,0.5,0.25,0.0}\lambda \in \{1.0, 0.75, 0.5, 0.25, 0.0\}, where λ=1.0\lambda=1.0 and λ=0.0\lambda=0.0 denote the pure exploitation setting and LA2-intra-play, respectively, and intermediate values sample both LA-order coplayers throughout training. Figure 23,24 illustrate constant-scheduled LARL performs comparably to linear-scheduled LARL from Figure 4, given properly tuned λ\lambda. Such empirical evidence strengthens the importance of training an agent with both same and adjacent LA-order coplayers.

(continued next)

AuthorsOpenReview ↗

Response to Reviewer 1j9Z (2/2)

(continuing)

Q2.3 Justification for sequential updates. The authors contrast aggregated group updates in COALA and suggest sequential updates.

For an ego agent ii, the aggregated update scheme optimizes the following objective θi←arg⁡max⁡θiEθj∼p(θj)[Ji(θi,θj)]\theta^{i} \leftarrow \arg\max_{\theta^{i}} \mathbb{E}_{\theta^{j} \sim p(\theta^{j})} [ J^{i}(\theta^{i}, \theta^{j}) ] where jj indicates a coplayer, and p(θj)p(\theta^{j}) denotes the distribution over different coplayers. This objective implicitly assumes no interference among gradient updates against different coplayers, which may not hold in general. For instance, the Half-Cheetah-Dir benchmark [4] contains two tasks that reward the agent for moving forward and backward, respectively. In another coordination matrix game as follows, if two different coplayers follow always-C and always-D, respectively, the ego agent's gradients optimizing towards both coplayers are misaligned. To this end, since we could not guarantee the no-gradient-interference generally holds in social dilemmas, we lift this assumption for generality and adopt the sequential update scheme from meta-RL literature [5].

CD
C1,10,0
D0,01,1

Q2.4 Remarks on difference from other methods. ... it is not clear to me whether it differs from standard per-trajectory methods.

To interpret LARL's novelty, we would like to clarify that our method is characterized not by its policy optimization approach, but by its data collection and training schemes. In a multi-agent system, agents interact and learn in a common environment, leading to inter-dependent decision-making and learning processes. Even when an agent unilaterally follows the same rollout and optimization strategy, different coplayers can lead to different learning outcomes. To understand the learning dynamics in such systems, we establish an formulation of learning awareness and LA-orders, upon which we propose LARL to further our understanding. We observe important findings not seen in prior work. In Sec 5.4 Figure 6, (a,e), (b,f), and (c,g) show similar behavior despite different groupings, demonstrating that LA-orders are relative and dynamic. The same LA2 agent, when trained with different coplayers -- LA1 in (a,e) or LA3 in (c,g) -- exhibit completely different behavior. Moreover, LARL enables multi-LA-order support, which presents LA3-vs-LA2-vs-LA1 and their interesting chain reaction in (d,h). These novel findings highlight the importance of our LA-order framework and LARL method in studying multi-agent learning dynamics.

Q3.

Q3 Remarks on future work. I think a promising future line of work is algorithmically applying the LA ranking to different types of information, similar to the discussions in Appendix A.2.

Thanks for pointing this out! We add the following discussion to the revised manuscript, Section 7.

This work aims to provide a general understanding of learning awareness and establish a relationship among asymmetric learning awareness information. Built upon our formulation, our method LARL is a natural model-free algorithm that supports multiple LA orders while seeking to minimize assumptions. As the reviewer pointed out, based on different assumptions, it is a promising direction for future work to explore learning awareness in the form of different types of information. Since yi,jy_{i,j} is defined as a set of information types, our framework naturally accommodates multiple information types (e.g., a mixture of gradients, interval-typed histories, and value functions), and investigating the resulting learning behaviors would also be an interesting direction. We also seek to explore implicit model-based approaches, and off-policy support for stronger sample efficiency.


[1] Hausknecht, Matthew J., and Peter Stone. "Deep recurrent Q-learning for partially observable mdps." AAAI fall symposia. Vol. 45. 2015.

[2] Ni, Tianwei, Benjamin Eysenbach, and Ruslan Salakhutdinov. "Recurrent model-free RL can be a strong baseline for many POMDPs." arXiv preprint arXiv:2110.05038 (2021).

[3] Lu, Christopher, et al. "Model-free opponent shaping." International Conference on Machine Learning. PMLR, 2022.

[4] Rakelly, Kate, et al. "Efficient off-policy meta-reinforcement learning via probabilistic context variables." International conference on machine learning. PMLR, 2019.

[5] Duan, Yan, et al. "Rl 2^ 2: Fast reinforcement learning via slow reinforcement learning." arXiv preprint arXiv:1611.02779 (2016).

Reviewer_KnDgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies learning-aware multi-agent reinforcement learning, where an agent accounts for how its coplayers learn and update their policies. It formalizes learning-awareness orders through nested sets of available learning information and proposes LARL, which uses an agent’s interaction history to capture distribution changes induced by coplayer learning. The method operationalizes different orders through recurrent state and different update schedules. Experiments cover iterated matrix games and state-based environments, including Cleanup, Coins, and Coop-Mining, and compare LARL with methods such as LOLA, MFOS, and COALA.

Critical strengths
  1. The paper places “how the coplayer learns” within a common outer decision framework. This provides a useful conceptual lens for comparing methods such as LOLA, MFOS, and COALA.
  2. LARL aims to infer learning-induced distribution changes from an agent’s own interaction history, without directly requiring the coplayer’s gradients or update rule. The evaluation spans matrix games and state-based tasks, and Table 6 reports higher average returns for LARL on Coins and Coop-Mining in the cooperative experiments.
Critical weaknesses
  1. The implemented learning-awareness order is not validated against the formal definition. Definition 3.2 orders agents by inclusion of their learning-information sets, but LARL primarily implements higher orders through longer, increasingly spaced update intervals, alongside recurrent state and grouped updates. The paper does not show that these choices provide the additional coplayer-learning information required by the definition. As a result, the experiments cannot isolate the effect of learning-awareness order from rollout length, GRU memory, or update grouping.
  2. The evidence does not support some broad claims, and key details needed to assess and reproduce the results are missing. The claim that experience-based methods “consistently exploit their coplayers” conflicts with the reported rΔr_\Delta values for LARL in Coop-Mining and Stag Hunt and with Appendix B.3’s statement that no method successfully exploits its coplayer in Coop-Mining and ISH. The cross-play results also do not establish that LARL is uniformly strongest. In addition, the paper does not specify the significance-testing procedure, comparison unit, multiple-comparison handling, or the full population-LARL auxiliary objective and its coefficient.
Questions for authors
  1. How does the update-interval schedule make higher-order LARL agents satisfy the larger learning-information sets in Definition 3.2? Could the authors provide controlled ablations, with matched network and update budgets, to separate the effects of update interval, cross-episode recurrent state, and grouped-update order?
  2. How should the claim that experience-based methods “consistently exploit” coplayers be reconciled with the Coop-Mining and Stag Hunt results in Table 3 and the statement in Appendix B.3? Please also clarify how cross-play averages and rΔr_\Delta are computed, and specify the statistical tests and uncertainty estimates used for the reported comparisons.
2 comments
AuthorsOpenReview ↗

Response to Reviewer KnDg (1/2)

Thank you for your insightful feedback. Your proposed ablation experiments helped clarify our contributions and strengthen our work.

In the updated revision, we marked updates in the main text in blue, and included new experiments and implementation details in Appendix G and Appendix I, respectively. We list related experiments in the following table for easy reference.

ExperimentsSectionsRemarks
LA1-intra-play, LA2-intra-play (new)Appendix H.3; Figure 25,26,27,31, Table 8equally-update-frequent agents trained together; ablate update interval and grouping
constant grouping schedule (new)Appendix H.2; Figure 23fixed probability of sampling lower-LA-order coplayer; emphasize importance of grouping with both LA-order coplayers
MLP vs RNN in LARL (new)Appendix H.1; Figure 20,21,22separate the effect of RNN hidden state; note the comparable performance of MLP-variant, showing generality of LARL
Cross-play (new)Appendix F.5additional results and clean summaries

*To avoid interference with figure/table numbering, we kept the original figures/tables in previous order and appended new ones to the end of the appendix.

Please kindly find our detailed responses below.

W1 & Q1.

Q1.1 LA formulation. How does the update-interval schedule make higher-order LARL agents satisfy the larger learning-information sets in Definition 3.2?

We appreciate the question. In Section 3.3, we used y~i,j\tilde{y}^{i,j} to denote information about jj's learning available to ii through its own history. It contributes the type HTj\mathsf{H}_{T_j} to yijy_{ij} only if it includes experience generated both before and after at least one update of coplayer jj (which is the case of higher-order/less-frequent-updating LARL agent ii vs. lower-order/more-frequent-updating LARL agent jj). Otherwise, it contributes no learning-awareness type. In this way, the ordering of agents in LARL can be regarded as a natural instantiation of Definition 3.2. Now, we update the manuscript accordingly (Section 4, lines 258-262, and Table 1) to make this connection more clear. Please also refer to our response to Reviewer jVY8 (Q1 and W1) for a detailed explanation.

Q1.2 Update interval and grouping. Could the authors provide controlled ablations, with matched network and update budgets, to separate the effects of update interval, cross-episode recurrent state, and grouped-update order?

First to clarify, in each agent group, both agents already have matched network configurations (see Section I.1 & Table 14 for hyperparameters). Both agents also use the same update budget, as they rollout the same number of transitions (Figure 2 exemplifies the same transition numbers despite different update intervals). To ablate the update interval and grouping, in Sec H.3 we present experiments on intra-play, where identically configured agents are paired to train. Figure 25 shows that intra-play, with either LA1/LA2 config, downperforms LARL in Coins, Prisoner's Dilemma, and Coop Mining, especially in Prisoner's Dilemma where it fails completely (Figure 27). Figure 31 and Table 8 also show that strong training-time performance in CleanUp does not guarantee strong ad-hoc cross-play performance. Moreover, ablations on grouping schedules (Figure 23) indicate that a constant grouping probability also performs well, given different LA-order coplayers are sampled to form groups. Both experiments underscore the importance of training agents with coplayers of different update frequencies and, consequently, different LA-orders.

Q1.3 Ablate network architecture. Could the authors provide controlled ablations ... cross-episode recurrent state ...?

We thank the reviewer for pointing out this potential source of confusion. First, we clarify the role of the RNN/GRU. The primary reason we adopt an RNN is to address the ubiquitous partial observability (PO) challenge in Deep RL [1,2], as we lift the full observability assumption for generality. The RNN is not an integral component of LARL but an implementation choice. Using the same architecture across all experiments ensures fair comparisons and minimizes tuning. In the updated revision, we moved related descriptions from Section 4 to Appendix I.2 to avoid potential confusion.

Then, to isolate the effect of cross-episode memory, we compare the RNN-based LARL to its MLP variant which completely eliminates the RNN's effect, in two nearly fully observable tasks, CleanUp and Coins (with only zapped_timer and step_count unobservable), and all matrix games. Figure 20,21,22 show that LARL-MLP performs similarly to LARL-RNN across tasks, quantitatively and qualitatively, isolating network architectures' effect for effective learning.

(continued next)

AuthorsOpenReview ↗

Response to Reviewer KnDg (2/2)

(continuing)

W2 & Q2.

Q2.1 Clarification on exploitation behavior. How should the claim that experience-based methods “consistently exploit” coplayers be reconciled with the Coop-Mining and Stag Hunt results in Table 3 and the statement in Appendix B.3?

Reviewer jVY8 similarly asked about exploitation guarantee in their W1 & Q1. A learning-aware method's exploitation behavior can be indicated by its stronger -- at least not weaker -- performance than its low LA-order coplayer. The performance gap is task-dependent, varying with task properties. In a pure cooperative task, for instance, a positive gap may not exist. In Table 3, experience-based methods achieve positive rΔr_\Delta's in most tasks, while the non-positive rΔr_\Delta's correspond to overlapping margins with their low-LA-order coplayer in Figure 8. Hence, they are consistently not worse than their coplayer.

As for Coop Mining and ISH, we discussed the phenomena raised by the reviewer in Appendix B.2 & B.3 (current Sec F.2 & F.3), and claim that these domains "may not pose a significant social learning challenge". The strong performance by LA2-intra-play also support this point.

Note: a method's exploitation gap rΔr_\Delta is computed by subtracting its mean reward by its coplayer's mean reward at the final iteration. We have also clarified this computation in Table 3's caption.

Q2.2 Clarification on cross-play results. The cross-play results also do not establish that LARL is uniformly strongest.

We provide the cross-play results in mean returns in Figure 28,29,30, and additional results in Figure 31,32,33 with large agent pools, with per-method statistics summarized in Table 8,9,10, respectively. We do not claim LARL is the strongest in every task, but it is the strongest in terms of overall performance.

The computation procedure for cross-play results is added to Sec H.4.

Q2.3 Remarks on statistical tests and hyperparameters. Please also clarify how cross-play averages and (r_\Delta) are computed, and specify the statistical tests and uncertainty estimates used for the reported comparisons.

In summarized results like Table 6, in each task, we perform two-sided t-tests with a significance threshold of 0.05 between the best-performing method and each other method. We accordingly updated Sec F.2 (previous Sec B.2) to include it. We incline to report the unaggregated learning results, e.g., Figure 4,8, as they provide a more comprehensive view of the training process.

Hyperparameters are summarized in Sec I.1 (Table 14-21), which specify the default settings across methods & tasks, and per-method overrides.


[1] Hausknecht, Matthew J., and Peter Stone. "Deep recurrent Q-learning for partially observable mdps." AAAI fall symposia. Vol. 45. 2015.

[2] Ni, Tianwei, Benjamin Eysenbach, and Ruslan Salakhutdinov. "Recurrent model-free RL can be a strong baseline for many POMDPs." arXiv preprint arXiv:2110.05038 (2021).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle message to Reviewers/AC

Dear Reviewers and AC,

We again appreciate the reviewers' valuable feedback that significantly improved our manuscript. We have carefully considered all comments and revised the manuscript accordingly. As the discussion phase approaches its end, we would welcome any further feedback, particularly on whether our responses and additional experiments have addressed the concerns.

We hereby summarize our responses, and highlight the major updates in the revised manuscript.

  1. Clarification on learning-awareness formulation (Reviewer KnDg's Q1.1, 1j9Z's Q1.1, jVY8's Q1). We strengthened the connection between LA-order and LARL in the response to each reviewer. We accordingly revised Section 3.3 to clarify the role and scope of LA information types, Section 4 to explicitly derive LARL's pairwise information types and Table 7 on how LARL corresponds to Section 3, and Table 1 to show type labels of existing approaches.
  2. New ablation experiments.
  • On recurrent net (Reviewer KnDg's Q1.3, 1j9Z's Q2.1, jVY8's Q2.2). The main reason for adopting an RNN is to tackle ubiquitous partial observability in Deep RL (Appendix I.2), and it is not required by the algorithm itself. New experiments compared LARL-RNN to LARL-MLP in Appendix H.1, and their comparable performance shows RNN is not the main source of LARL's effectiveness.
  • On coplayer sampling schedule (Reviewer KnDg's Q1.2, 1j9Z's Q2.2, jVY8's Q2.3). New experiments in Appendix H.2 sweeps constant probabilities of sampling coplayers. Results show that a constant schedule performs comparably to the linear schedule, and underscore the importance of training with both same and adjacent LA-order coplayers.
  1. New intra-play baselines (Reviewer KnDg's Q1.2, jVY8's Q2.4). We present intra-play -- i.e., self-play without parameter sharing -- experiments in Appendix H.3. The negative results in IPD underscore the importance of training agents with coplayers of different LA-orders.
  2. Supplementary cross-play results (Reviewer jVY8's Q3.1). We first clarify the interpretation of existing cross-play results. We then provide additional results in Appendix H.4. LARL achieves the strongest overall performance.
  3. Clarification on exploitation behavior (Reviewer KnDg's Q2.1, jVY8's Q1). We clarify exploitation as an empirical behavior where the learning-aware method's performance is at least not weaker than its low LA-order coplayer. We note that while LARL exhibits exploitation, the performance gap is task-dependent.
  4. Discussions on novelty (Reviewer 1j9Z's Q2.4, jVY8's Q2.1). We clarify that our method is characterized not only by its policy optimization approach, but also by its data collection and training schemes. We formalize learning awareness and LA-orders, and propose LARL to further our understanding. Empirical evidence in Sec 5.4 & F.4 highlights the relativeness of LA-order, and mutual influences on learning dynamics over adjacent LA-orders.
  5. Justification for sequential updates (Reviewer 1j9Z's Q2.3). The aggregated gradient update scheme implicitly assumes no conflicting objectives associated with different coplayers, which may not hold in general. We therefore lift this assumption and adopt the sequential scheme from meta-RL literature.
  6. Implementation details and hyperparameters (Reviewer KnDg's Q2.3, 1j9Z's Q1.3, jVY8's Q3.2). Sec I.1 provides the complete tuning procedure and hyperparameters.
  7. Discussions on future work (Reviewer 1j9Z's Q3). We extended our discussion on future work in Sec 7.

Best Regards,

Authors of #54087

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.