- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper addresses the challenges of temporal and multi-agent credit assignment in cooperative multi-agent RL within sparse-reward environments. The main contribution of the paper is the use of an implicit reward derived from a shared Q/V Bradley-Terry model. Each agent's dense local reward is defined using the inverse Bellman equation with Q/V obtained from the BT model. Additionally, the paper presents an identity that characterizes the discrepancy between local and team gradients under GAE. Empirical evaluations of the proposed method are conducted in sparse-reward settings across three benchmark domains: SMACv2, MAMuJoCo, and SMACLite.
Critical strengths
-
The use of Bradley–Terry model is a clean and well-motivated design choice, as it treats episodes as pairwise comparisons based on observed terminal outcomes. This approach transforms sparse cumulative reward signals into a differentiable trajectory score objective. Then, using the fitted Q/V heads from BT model to compute the implicit local reward using the inverse Bellman equation provides a principled bridge from terminal feedback to dense per-step credit for each agent.
-
The exact GAE identity in Eq. 8 is a useful structural guarantee because the normalized mixer weights sum to one. Mixing the local TD residuals yields the global residual, and the common GAE recursion preserves that equality across all time steps.
-
The empirical evaluation is thorough and well-structured, evaluated on three benchmark problems with sparse reward setting. The proposed method consistently outperforms the baselines used.
Critical weaknesses
-
Multi-agent credit assignment problem: Given that the main problem setting of the paper is cooperative. It is not clear how the proposed method specifically addresses the multi-agent credit assignment problem. The BT model is mainly about fitting the trajectory score from the episode outcome and using it in the inverse-Bellman residuals, which spread the fitted score across time. That is precisely temporal credit assignment. Eq. 1 and Eq. 8 are the equations that intuitively address the agent credit assignment. However, Eq. 1 is a known linear decomposition method, and Eq. 8 is a trivial consequence of Eq. 1 and Eq. 3. The multi-agent credit assignment problem is not adequately addressed in the paper, which is the key challenge in cooperative MARL.
-
The baseline family is narrow, everything compared is a PPO/delayed-reward variant, not the standard cooperative MARL methods such as VDN/QMIX/QPLEX/COMA that addresses the agent credit assignment problem. COMA is mentioned in the related work but not compared with.
-
Many empirical results are deferred to the appendix, which noticeably hampers readability and assessability. This weakens the narrative flow and makes it difficult to judge the strength of the evidence on first reading. I would strongly recommend relocating the key tables and figures into the main text.
Questions for authors
-
Clarification on : intuition behind adding to reward is not clear because is a global signal whereas is local.
-
Clarification on why use SMACLite since SMACv2 is already used?
Awaiting Your Feedback on Our Rebuttal
Dear Reviewer eyj4,
Thank you again for your review. As the discussion period is drawing to a close, we would appreciate it if you could take a look at our rebuttal. We would be happy to clarify any points or address any remaining concerns while the discussion is still open.
Best regards,
The Authors