Learning Awareness for Effective and Efficient Learning in Social Dilemmas

Many real-world multi-agent systems involve social dilemmas, where agents have partially aligned but conflicting interests, and mutual cooperation can benefit agents in long run. Social dilemmas are challenging, as self-interested agents often choose strategies that are suboptimal, particularly when other agents are also learning. Recent learning-aware methods offer a promising direction, but often make restrictive assumptions about how agents understand the coplayers' learning processes. In this work, we conceptualize learning as action within a decision model to help characterize learning awareness (LA). We accordingly introduce the learning awareness order (LA-order) over learning agents, where an LA(k+1) agent can "exploit" an LAk coplayer by leveraging its information about the coplayer's learning process. The LA-order naturally provides a lens to understand a wide spectrum of learners, ranging from fixed random agents (LA0), self-interested naive learners LA1, to prior learning-aware agents (LA2${}^{+}$). Based on this understanding, we introduce a model-free, decentralized multi-agent reinforcement learning (MARL) approach extensible to any LA-orders, which performs learning-aware RL through dynamic grouping of agents, with each agent leveraging solely its own experience. Through evaluation across diverse social dilemma challenges, in both iterated matrix games and state-based benchmarks, we demonstrate the effectiveness of our algorithms, including under ad-hoc generalization, and the value of the LA-order. Experiments also show that agents with adjacent LA-orders mutually influence each other's learning dynamics.