- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed MMPS method which is a reward weight tuning method which takes a multi-objective reinforcement learning perspective to tune a fixed library of auxiliary rewards for a given task. MMPS trains a shared, weight-conditioned actor–critic across reward mixtures and uses its predicted primary-task value to select the mixture for each rollout. MMPS achieves higher task performance than existing weight-tuning methods under matched environment-interaction budgets, with its largest gains on difficult manipulation tasks, achieving 2.26× the score of policies trained with expert-tuned reward weights.
Critical strengths
-
The empirical performance is strong which outperforms BO, PBT and BiPaRS
-
The framework is simple and easy to implement.
Critical weaknesses
-
The key component of proposed method relies how can task critic can accurate rank the candidate reward weights. While the paper do evaluate random select vs MMPS, the random select itself is a weak baseline. The result do not directly validate wheather it can rank weight correctly. As a result, it remains unclear whether the gains stem from accurate task-directed weight selection or from the richer reward-conditioned actor-critic architecture and multi-head value-function representation.
-
For the theoretical part, in Theorem J1, auther frezz the selector which select same weight did not really reflect the actual method. In actual method, it can choose different weight every rollout and updates. The objective may becoming a running target. The theory therefore provides limited insight into the stability and convergence behavior of the algorithm used in practice.
Questions for authors
-
Line 78, auther cite the "... are auxiliary shaping rewards available during training (Ng et al., 1999). " However, the Ng 1999 mainly talks about shaped reward in PBRS format which is not related with the reward type of auther proposed in this paper. Can auther clearify why the PBRS reward shaping format is mentioned here? or what is the relationship between proposed method with PBRS?
-
The proposed method required much larger network ("MMPS agent has approximately 0.81M–1.12M") compare withe default PPO baseline (" 0.13M–0.25M "). Can auther use parallel critic to bring default PPO the same scale at MMPS and compare?
-
Can auther provide analysis on how is accurate?