- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses automatic reward shaping in reinforcement learning (RL). In contrast to handcrafted rewards, which induce designer bias, i.e., pre-specifying useful behavior while restricting potential informative behavior for training, the paper proposes BLUR to learn a dense reward function for easy learning. It compresses the experience trajectories into latent representations to cluster behavioral patterns. It then assigns performance values to each cluster to learn a reward model that can associate between behavioral patterns and the actual task performance. BLUE is evaluated on cooperative multi-agent tasks and compared with standard baselines, such as MAPPO, as well as a re-implementation of SASR, an adaptive reward shaping approach for single-agent RL tasks.
Critical strengths
The paper addresses an important and long-standing problem in reinforcement learning, where sparse rewards can lead to suboptimal learning due to lacking exploration. Handcrafting rewards is tedious and can yield unintended side effects, which needs to be addressed adequately.
Critical weaknesses
- Clarification of the setting:
- Overall, the paper addresses a single-agent RL problem by focusing on a single scalar reward, without considering any agent interactions, conflicts, or credit assignments. However, it uses multi-agent environments to evaluate its approach with multi-agent benchmarks (SMAC) and baselines (MAPPO) rather than related single-agent work (SIL, RUDDER, etc.).
- Although it briefly states that multi-agent tasks could be collapsed to single-agent problems, the reverse does not hold: Having several self-interested agents could lead to emergent dilemmas, which cannot be addressed trivially by using single-agent methods [1]. Thus, I suggest a clearer focus toward single-agent MDPs instead of Dec-POMDPs, which seems like an over-design to me.
- Furthermore, SMAC is noted to be outdated [2].
- Despite the paper's criticsm against handcrafted rewards, the approach itself requires domain-dependent design, e.g., to define the statistics necessary for (Equation 1), which is necessary to conduct supervised reward learning (Equation 5). Thus, it does not seem to solve the actual issue of handcrafting.
- Evaluation
- The paper only reports the final test performance of BLUR in a table. Figure 5 reports the training curves compared with handcrafted rewards. However, I do not see any substantial advantage of BLUE, i.e., evidence about how it overcomes the designer bias mentioned above.
- Due to the actual single-agent RL focus, the multi-agent evaluation appears misleading. The underperformance of SASR might be confounded by multi-agent aspects (e.g., credit assignment, which could affect the single-agent reward adaptation). Conversely, other adaptive reward methods that the paper did not compare with could still outperform BLUE (either in single-agent or multi-agent settings). Due to the missing experimental comparison, I cannot confirm the significance of the results.
[1] Leibo et al., "Multi-agent Reinforcement Learning in Sequential Social Dilemmas", AAMAS-17
[2] Ellis et al., "SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning", NeurIPS Benchmarks 2023
Questions for authors
- What is meant by “consequence” regarding (Equation 2)? How is it represented?
- Why does the auxiliary performance-prediction head require pooling of all hidden states (Equation 4)? should already contain all preceding information, according to the general RNN design.