- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper extends the notion of using test driven reward design for RL to MARL. It is a good contribution to the MARL field since Test-Driven paradigm is something that needs to be tested for its application to MARL. Although it is an incremental contribution in novelty, I still find that it is a necessary incremental contribution that has merit. The idea is built on both off policy and on policy based MARL backbones which is important to have for real applicability and adaptability. The experiment environments cover the expectations (having LBF MPE and SMAC) to have sufficient experimental verification for cooperative MARL work in this domain. The experiments, ablations and behavioral studies are carried out neatly. The code of this work is open sourced.
Critical strengths
- Extensive experimentation on MPE, LBF and SMAC along with relevant ablations and behavioral studies.
- Theoretical analysis for extending the test driven paradigm to MARL (although based on impractical assumptions)
- I like the engineering behind to create pass fail and indicative tests per environment along with the adaptations to make it work in MARL setup
Critical weaknesses
- The theoretical analysis is based on impractical assumptions and as far as I could see (I might have missed in that case please let me know) there is no theoretical analysis on how close of a bound we can get using the test driven paradigm to approximate sound dense rewards (or at least the measure of how aligned learned rewards would be with the true game rewards)
- Return learning and return decomposition is a total black box. The behavioral analysis provided in the appendix is informative, but it is not enough for me to be convinced this black box can always learn sensible rewards with this algorithm.
- The experimentation is comprehensive so I do expect some performance divergence between the baselines. But from the results, I can not say that TD method consistently increases performanse with respect to QMIX-MAPPO and in many cases their dense counter parts look better.
Questions for authors
- Why do you think it is easier / better to engineer test cases for MARL than just engineering dense rewards. Reward engineering has its own difficulties and limits, but just using a black box to learn rewards from test case results doesn't seem to be trustworthy and according to the plots it doesn't perform better either (at least consistently).
- Do you think there can be any theoretical work / analysis on directly approximating rewards from test results (without ad hoc assumptions)?
- Do you have any behavioral analysis etc on why the effect of TD differs across environments? Can it be due to inferior test case design? Since it looks like the module is a black box, and the results supporting the effectiveness of TD MARL paradigm is not consistent I would really like to know what kind of environments/games are more suitable to use the TD paradigm vs dense rewards or the other baselines.
- The main thing I would like to see in this paper is why and when TD paradigm overperforms and underperforms in specific experiments and how we can use this to determine when it is a good time to use TD paradigm.
4 comments
Thank you for the detailed explanations. Many of my concerns are addressed. The ones remaining, that also lead me to keep my score, is the following. There is a black box learning process for rewards which is only convincing when the results are good, the test driven approach doesn't consistently over perform dense reward etc, and I don't see (if I didn't miss anything) an example experiment where test driven approach is a must or a big help. In this case, I cannot be convinced from this paper that test driven reward design for marl is a useful and necessary toolset. Are there any other things I should be aware of and consider?
Part 1: Why Learn a Reward from Test?
Thank you for explaining what still gives you pause. We understand your concern that learning a reward adds complexity, and that final scores alone do not establish when that complexity is worthwhile. We appreciate the opportunity to clarify why we started from task requirements and how we checked what TdMARL learned.
-
Why use TdMARL when a dense reward can already solve the task? Our starting point is how a team’s success is judged. In Cooperative Navigation, for example, the team must sustain landmark coverage and avoid collisions. If both are required, better coverage cannot compensate for a collision. A designer may know exactly how to check each requirement while having little basis for deciding how much reward to assign at each step. If no dense reward is available, a dense-reward baseline has no training signal for comparison. If one can be engineered, doing so still requires domain knowledge to select intermediate events, value them, and tune their relative weights. We therefore evaluated TdMARL across settings with terminal feedback, established dense rewards, and additional behavioral requirements. These settings let us examine what a designer can achieve by specifying success directly.
-
How can we assess a learned reward beyond the final score? Interpretability is a broader challenge in deep learning. Inverse RL has long studied how to infer rewards from behavior [1], with applications to drivers’ route choices [2] and robotic manipulation [3]. TdMARL belongs to this broader family of reward-learning methods. Its supervision comes from preferences over team trajectories generated by explicit task tests. This gives us two concrete checks. Figure 4 shows that the return model ranks wins above losses in 88.7–96.3% of sampled pairs and makes the ordering errors visible. Figures 6–7 let us inspect the stepwise reward in one LBF rollout: the agents receive positive feedback as they approach the fruit together, and the reward peaks at pickup. Thus, readers can examine both how the model judges a trajectory and which steps it rewards within it.
References
[1] Ng, A. Y., & Russell, S. (2000, June). Algorithms for inverse reinforcement learning. In Icml(Vol. 1, No. 2, p. 2).
[2] Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2008, July). Maximum entropy inverse reinforcement learning. In AAAI(Vol. 8, pp. 1433-1438).
[3] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning (pp. 49-58). PMLR.
Part 2: What Does TdMARL Offer in Practice?
-
What does TdMARL contribute to cooperative MARL? Our goal was to give designers another way to train a team: specify what a successful joint trajectory should look like and learn a reward from those requirements. We chose QMIX and MAPPO to test whether this idea works across two different approaches to cooperative learning. We agree that engineered dense rewards perform better in several settings, as our paper reports. The contribution is that designers can start from the criteria they use to judge the team, even when they do not know how to translate those criteria into rewards for individual actions.
-
When would a practitioner want this option? Suppose a delivery contract requires at least 80% of orders to arrive on time and at least 70% of vehicles to finish each shift above a specified battery reserve. Both thresholds must be met. The operator can check them at the end of a shift, yet the contract gives little guidance on how much reward a vehicle should receive for accepting another order, taking a detour, or recharging early. With TdMARL, the observed percentages measure progress, and the contractual thresholds define the tests. The operator specifies what makes a shift successful, and TdMARL learns how to reward individual steps.
Our primary goal has been to broaden the reward-design choices available to the MARL community, rather than to pursue state-of-the-art performance on every benchmark. We report where engineered dense rewards perform better and have made our code, tests, and experimental environments openly available so others can evaluate and build on TdMARL.
https://anonymous.4open.science/r/TdMARL-4FBB
Thank you again for asking what a practitioner would gain from TdMARL. We hope this clarifies where TdMARL offers practical value. If this addresses your remaining concern, we would be grateful if you would take it into account and consider raising your rating.
Reward Reliability and Design Benefits
Thank you for your thoughtful feedback. We have marked revisions in blue in the updated PDF.
W1/Q2: Can tests support reward-approximation theory?
Proposition 1 (p. 4) bounds test failure under separation and realizability assumptions. We add a trajectory-return analysis. Let Genv be environment return, G^ϕ the learned reward sum, and C(x)=ax+b, a>0, a fixed calibration. Using the decomposition error ϵπ from Eq. 7 (p. 4), the triangle inequality gives
Eπ∣C(G^ϕ)−Genv∣≤aϵπ+Eπ∣C(R^ψ)−Genv∣.
This separates decomposition error from alignment with environment return without Proposition 1's assumptions. Both terms are measurable on the evaluated policy's trajectories. Lines 193–199 (p. 4). Extending it to individual step rewards requires addressing identifiability, since different allocations can yield the same return. Further theory can identify tests and trajectory coverage that constrain reward recovery. Figures 8–10 (pp. 29–30) evaluate policies under the original environment rewards.
W2/Q1: What supports the reliability of learned rewards?
We understand this concern. TdMARL shares the motivation of inverse and preference-based RL, where learned rewards support robotic control and game playing (Finn et al., 2016; Fu et al., 2018; Christiano et al., 2017). We evaluate each stage and the full system.
Figure 4 (p. 9) reports correct ordering in 88.7–96.3% of sampled win–loss pairs, with 256 trajectories per panel. Figures 6–7 (p. 24) relate positive step rewards to cooperative approaches and pickups in a representative LBF rollout. Table 9 (p. 35) reports rollout diagnostics for separation and reconstruction. See lines 443–448 (p. 9), 1816–1831 (p. 34). Section 5.2 (pp. 7–9) and Appendix H (pp. 29–37) evaluate QMIX and MAPPO policies across MPE, LBF and SMACv2 using task metrics and environment returns.
W3/Q1: Does TdMARL consistently improve performance?
Dense-reward baselines perform better in several settings. TdMARL starts from the criteria used to judge task success and uses them to guide reward construction and learning. Our QMIX and MAPPO comparisons assess this design paradigm. The main text emphasizes task metrics. Appendix H.1 (pp. 29–31) reports standard environment returns. Improving performance and stability remains a research priority.
Q1: Why can test design be simpler?
In SMACv2, designers specify victory and targets for allied survival and enemy elimination. Our tester evaluates completed battles through terminal checks and trajectory averages of these counts (lines 1422–1443, p. 27).
Dense rewards also require valuing intermediate combat events. The official SMACv2 reward combines health/shield damage, kill and victory bonuses, weights and normalization (Ellis et al., 2023; lines 1444–1456, p. 27). TdMARL learns step rewards from outcome evaluations, reducing domain-specific shaping rules and manual balancing of reward terms. The same tests support QMIX and MAPPO.
Q3: What explains the observed behavioral differences?
Behavioral evidence. Lines 363–402 (pp. 7–8) and Figures 12–17 (pp. 31–33) show stable landmark assignments and encircling pursuit with Td-MAPPO, while Td-QMIX exhibits unstable assignments and less coordinated pursuit.
Test design and backbone. With the same CN tester, Td-MAPPO achieves coverage of 0.55, while QMIX and Td-QMIX reach 0.11 and 0.03. Figures 13–14 (pp. 31–32) illustrate the failures in our supplementary videos. QMIX leaves one landmark uncovered, while Td-QMIX moves without maintaining stable coverage. Thus, the tests support successful coordination, and the contrasting outcomes suggest backbone dependence. Lines 1719–1727 (p. 32).
Additional requirements. LBF shows how pairing requirements can shape behavior at comparable collection performance (lines 452–459, p. 9, and 1941–1943, p. 36). This offers a way to extend tasks with established rewards.
Q4: When is TdMARL a suitable choice?
SMACv2 tests provide survival and elimination guidance missing from terminal-only rewards, helping explain gains over sparse baselines. Dense rewards additionally encode health and shield damage, whose finer feedback may explain their advantage in some settings. In CN, Td-QMIX's failure to maintain landmark assignments accompanies its low coverage. Lines 363–375 (p. 7), 409–421 (p. 8), 452–459 (p. 9).
These observations motivate TdMARL when task criteria are clear but effective dense feedback is difficult to design, potentially including traffic scheduling with waiting-time and throughput targets. For mature rewards, LBF illustrates adding requirements. We would use these design needs to guide adoption, then assess performance with the chosen backbone.
Please do not hesitate to ask further questions. If we have addressed your concerns, please consider raising your rating.