acceptodds
Under review as a conference paper at ICLR 2027

Test-Driven Reward Design for Cooperative Multi-Agent Reinforcement Learning

Abstract

In reinforcement learning (RL), especially in multi-agent reinforcement learning (MARL), reward design remains a fundamental challenge, as dense rewards often require task-specific engineering, while sparse rewards provide insufficient learning guidance for effective coordination. We observe that task completion can be characterized by whether agents satisfy task-level behavioral requirements, which can serve as implicit tests for evaluating cooperative behaviors. Based on this insight, we adapt the paradigm of Test-Driven Reinforcement Learning (TdRL) to the multi-agent setting and develop **T**est-**D**riven **M**ulti-**A**gent **R**einforcement **L**earning (TdMARL), a framework that constructs return functions from task-level evaluations and derives informative reward signals for the multi-agent system, enabling a more intuitive and systematic design process for complex cooperative environments. We provide theoretical analysis establishing the validity of extending the test-driven paradigm from single-agent RL to cooperative MARL. Furthermore, we introduce MARL-specific adaptations to TdRL, including tailored data sampling and preference-evaluation mechanisms, together with a soft uniform prior for stable return-to-reward decomposition, to better accommodate the sensitivity and non-stationarity of multi-agent systems. Experiments across dense reward, sparse reward, and sparse reward with auxiliary signals demonstrate that TdMARL provides effective reward guidance and achieves robust coordination performance across diverse reward settings. TdMARL offers a new perspective on reward design for cooperative MARL by shifting the focus from manually engineering scalar rewards to specifying task-level behavioral requirements.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Test-Driven Reinforcement Learning, Reward Design, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FioEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends the notion of using test driven reward design for RL to MARL. It is a good contribution to the MARL field since Test-Driven paradigm is something that needs to be tested for its application to MARL. Although it is an incremental contribution in novelty, I still find that it is a necessary incremental contribution that has merit. The idea is built on both off policy and on policy based MARL backbones which is important to have for real applicability and adaptability. The experiment environments cover the expectations (having LBF MPE and SMAC) to have sufficient experimental verification for cooperative MARL work in this domain. The experiments, ablations and behavioral studies are carried out neatly. The code of this work is open sourced.

Critical strengths
  • Extensive experimentation on MPE, LBF and SMAC along with relevant ablations and behavioral studies.
  • Theoretical analysis for extending the test driven paradigm to MARL (although based on impractical assumptions)
  • I like the engineering behind to create pass fail and indicative tests per environment along with the adaptations to make it work in MARL setup
Critical weaknesses
  • The theoretical analysis is based on impractical assumptions and as far as I could see (I might have missed in that case please let me know) there is no theoretical analysis on how close of a bound we can get using the test driven paradigm to approximate sound dense rewards (or at least the measure of how aligned learned rewards would be with the true game rewards)
  • Return learning and return decomposition is a total black box. The behavioral analysis provided in the appendix is informative, but it is not enough for me to be convinced this black box can always learn sensible rewards with this algorithm.
  • The experimentation is comprehensive so I do expect some performance divergence between the baselines. But from the results, I can not say that TD method consistently increases performanse with respect to QMIX-MAPPO and in many cases their dense counter parts look better.
Questions for authors
  • Why do you think it is easier / better to engineer test cases for MARL than just engineering dense rewards. Reward engineering has its own difficulties and limits, but just using a black box to learn rewards from test case results doesn't seem to be trustworthy and according to the plots it doesn't perform better either (at least consistently).
  • Do you think there can be any theoretical work / analysis on directly approximating rewards from test results (without ad hoc assumptions)?
  • Do you have any behavioral analysis etc on why the effect of TD differs across environments? Can it be due to inferior test case design? Since it looks like the module is a black box, and the results supporting the effectiveness of TD MARL paradigm is not consistent I would really like to know what kind of environments/games are more suitable to use the TD paradigm vs dense rewards or the other baselines.
  • The main thing I would like to see in this paper is why and when TD paradigm overperforms and underperforms in specific experiments and how we can use this to determine when it is a good time to use TD paradigm.
4 comments
AuthorsOpenReview ↗

Reward Reliability and Design Benefits

Thank you for your thoughtful feedback. We have marked revisions in blue in the updated PDF.

W1/Q2: Can tests support reward-approximation theory?

Proposition 1 (p. 4) bounds test failure under separation and realizability assumptions. We add a trajectory-return analysis. Let GenvG_{\mathrm{env}} be environment return, G^ϕ\hat G_\phi the learned reward sum, and C(x)=ax+bC(x)=ax+b, a>0a>0, a fixed calibration. Using the decomposition error ϵπ\epsilon_\pi from Eq. 7 (p. 4), the triangle inequality gives

Eπ∣C(G^ϕ)−Genv∣≤aϵπ+Eπ∣C(R^ψ)−Genv∣.\mathbb E_\pi|C(\hat G_\phi)-G_{\mathrm{env}}|\le a\epsilon_\pi+\mathbb E_\pi|C(\hat R_\psi)-G_{\mathrm{env}}|.

This separates decomposition error from alignment with environment return without Proposition 1's assumptions. Both terms are measurable on the evaluated policy's trajectories. Lines 193–199 (p. 4). Extending it to individual step rewards requires addressing identifiability, since different allocations can yield the same return. Further theory can identify tests and trajectory coverage that constrain reward recovery. Figures 8–10 (pp. 29–30) evaluate policies under the original environment rewards.

W2/Q1: What supports the reliability of learned rewards?

We understand this concern. TdMARL shares the motivation of inverse and preference-based RL, where learned rewards support robotic control and game playing (Finn et al., 2016; Fu et al., 2018; Christiano et al., 2017). We evaluate each stage and the full system.

Figure 4 (p. 9) reports correct ordering in 88.7–96.3% of sampled win–loss pairs, with 256 trajectories per panel. Figures 6–7 (p. 24) relate positive step rewards to cooperative approaches and pickups in a representative LBF rollout. Table 9 (p. 35) reports rollout diagnostics for separation and reconstruction. See lines 443–448 (p. 9), 1816–1831 (p. 34). Section 5.2 (pp. 7–9) and Appendix H (pp. 29–37) evaluate QMIX and MAPPO policies across MPE, LBF and SMACv2 using task metrics and environment returns.

W3/Q1: Does TdMARL consistently improve performance?

Dense-reward baselines perform better in several settings. TdMARL starts from the criteria used to judge task success and uses them to guide reward construction and learning. Our QMIX and MAPPO comparisons assess this design paradigm. The main text emphasizes task metrics. Appendix H.1 (pp. 29–31) reports standard environment returns. Improving performance and stability remains a research priority.

Q1: Why can test design be simpler?

In SMACv2, designers specify victory and targets for allied survival and enemy elimination. Our tester evaluates completed battles through terminal checks and trajectory averages of these counts (lines 1422–1443, p. 27).

Dense rewards also require valuing intermediate combat events. The official SMACv2 reward combines health/shield damage, kill and victory bonuses, weights and normalization (Ellis et al., 2023; lines 1444–1456, p. 27). TdMARL learns step rewards from outcome evaluations, reducing domain-specific shaping rules and manual balancing of reward terms. The same tests support QMIX and MAPPO.

Q3: What explains the observed behavioral differences?

  1. Behavioral evidence. Lines 363–402 (pp. 7–8) and Figures 12–17 (pp. 31–33) show stable landmark assignments and encircling pursuit with Td-MAPPO, while Td-QMIX exhibits unstable assignments and less coordinated pursuit.

  2. Test design and backbone. With the same CN tester, Td-MAPPO achieves coverage of 0.55, while QMIX and Td-QMIX reach 0.11 and 0.03. Figures 13–14 (pp. 31–32) illustrate the failures in our supplementary videos. QMIX leaves one landmark uncovered, while Td-QMIX moves without maintaining stable coverage. Thus, the tests support successful coordination, and the contrasting outcomes suggest backbone dependence. Lines 1719–1727 (p. 32).

  3. Additional requirements. LBF shows how pairing requirements can shape behavior at comparable collection performance (lines 452–459, p. 9, and 1941–1943, p. 36). This offers a way to extend tasks with established rewards.

Q4: When is TdMARL a suitable choice?

SMACv2 tests provide survival and elimination guidance missing from terminal-only rewards, helping explain gains over sparse baselines. Dense rewards additionally encode health and shield damage, whose finer feedback may explain their advantage in some settings. In CN, Td-QMIX's failure to maintain landmark assignments accompanies its low coverage. Lines 363–375 (p. 7), 409–421 (p. 8), 452–459 (p. 9).

These observations motivate TdMARL when task criteria are clear but effective dense feedback is difficult to design, potentially including traffic scheduling with waiting-time and throughput targets. For mature rewards, LBF illustrates adding requirements. We would use these design needs to guide adoption, then assess performance with the chosen backbone.

Please do not hesitate to ask further questions. If we have addressed your concerns, please consider raising your rating.

Thank you for the detailed explanations. Many of my concerns are addressed. The ones remaining, that also lead me to keep my score, is the following. There is a black box learning process for rewards which is only convincing when the results are good, the test driven approach doesn't consistently over perform dense reward etc, and I don't see (if I didn't miss anything) an example experiment where test driven approach is a must or a big help. In this case, I cannot be convinced from this paper that test driven reward design for marl is a useful and necessary toolset. Are there any other things I should be aware of and consider?

AuthorsOpenReview ↗

Part 1: Why Learn a Reward from Test?

Thank you for explaining what still gives you pause. We understand your concern that learning a reward adds complexity, and that final scores alone do not establish when that complexity is worthwhile. We appreciate the opportunity to clarify why we started from task requirements and how we checked what TdMARL learned.

  1. Why use TdMARL when a dense reward can already solve the task? Our starting point is how a team’s success is judged. In Cooperative Navigation, for example, the team must sustain landmark coverage and avoid collisions. If both are required, better coverage cannot compensate for a collision. A designer may know exactly how to check each requirement while having little basis for deciding how much reward to assign at each step. If no dense reward is available, a dense-reward baseline has no training signal for comparison. If one can be engineered, doing so still requires domain knowledge to select intermediate events, value them, and tune their relative weights. We therefore evaluated TdMARL across settings with terminal feedback, established dense rewards, and additional behavioral requirements. These settings let us examine what a designer can achieve by specifying success directly.

  2. How can we assess a learned reward beyond the final score? Interpretability is a broader challenge in deep learning. Inverse RL has long studied how to infer rewards from behavior [1], with applications to drivers’ route choices [2] and robotic manipulation [3]. TdMARL belongs to this broader family of reward-learning methods. Its supervision comes from preferences over team trajectories generated by explicit task tests. This gives us two concrete checks. Figure 4 shows that the return model ranks wins above losses in 88.7–96.3% of sampled pairs and makes the ordering errors visible. Figures 6–7 let us inspect the stepwise reward in one LBF rollout: the agents receive positive feedback as they approach the fruit together, and the reward peaks at pickup. Thus, readers can examine both how the model judges a trajectory and which steps it rewards within it.

References

[1] Ng, A. Y., & Russell, S. (2000, June). Algorithms for inverse reinforcement learning. In Icml(Vol. 1, No. 2, p. 2).

[2] Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2008, July). Maximum entropy inverse reinforcement learning. In AAAI(Vol. 8, pp. 1433-1438).

[3] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning (pp. 49-58). PMLR.

AuthorsOpenReview ↗

Part 2: What Does TdMARL Offer in Practice?

  1. What does TdMARL contribute to cooperative MARL? Our goal was to give designers another way to train a team: specify what a successful joint trajectory should look like and learn a reward from those requirements. We chose QMIX and MAPPO to test whether this idea works across two different approaches to cooperative learning. We agree that engineered dense rewards perform better in several settings, as our paper reports. The contribution is that designers can start from the criteria they use to judge the team, even when they do not know how to translate those criteria into rewards for individual actions.

  2. When would a practitioner want this option? Suppose a delivery contract requires at least 80% of orders to arrive on time and at least 70% of vehicles to finish each shift above a specified battery reserve. Both thresholds must be met. The operator can check them at the end of a shift, yet the contract gives little guidance on how much reward a vehicle should receive for accepting another order, taking a detour, or recharging early. With TdMARL, the observed percentages measure progress, and the contractual thresholds define the tests. The operator specifies what makes a shift successful, and TdMARL learns how to reward individual steps.

Our primary goal has been to broaden the reward-design choices available to the MARL community, rather than to pursue state-of-the-art performance on every benchmark. We report where engineered dense rewards perform better and have made our code, tests, and experimental environments openly available so others can evaluate and build on TdMARL.

https://anonymous.4open.science/r/TdMARL-4FBB

Thank you again for asking what a practitioner would gain from TdMARL. We hope this clarifies where TdMARL offers practical value. If this addresses your remaining concern, we would be grateful if you would take it into account and consider raising your rating.

Reviewer_qdJcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The idea is to specify what a team should do through trajectory-level tests, rather than hand-design every step's reward. The tests produce preferences between joint trajectories. A return model learns from those preferences, then a reward model turns the return into per-step team rewards for QMIX or MAPPO. The authors add targeted pair sampling, changes to the preference rules, and a soft uniform prior to keep reward decomposition stable.

Critical strengths
  • Describing what a good trajectory should look like can be easier than choosing step-wise reward weights. This gives a concrete way to train from those descriptions.
  • The authors test two different MARL backbones and several environments, rather than showing one successful combination. The larger-team experiments are useful too.
  • The uniform prior has a large effect in the ablations. It seems to fix a real training problem, not just give a small extra gain.
Critical weaknesses
  1. The comparison I'd most like to see is missing. In sparse SMACv2, the plain learner gets terminal outcomes, but the tester also uses average allied survival and enemy elimination over the trajectory. That's useful extra supervision. Why not turn those same indicators into a simple reward and compare against that? Right now I can't tell how much the two-stage learning adds beyond having those indicators in the first place.
  2. The extension from TdRL feels fairly incremental. The sampling changes, preference rules, and prior help, and the ablations support that. But I'd like a clearer explanation of what is specifically hard about the multi-agent case and why these changes amount to more than useful fixes to the existing pipeline.
  3. The theorem assumes test separation, an ideal successful policy that can be represented, and small decomposition and optimization errors. It tells us what follows if those hold, but not whether the learned models get there. I'm also puzzled that preference labels use pass-fail outcomes while the return model only sees indicative outcomes. If those inputs don't determine the pass-fail result, some of the labels may be impossible for it to fit.
  4. The uniform prior makes decomposition more stable, but that doesn't show the reward goes to the right actions. Spreading return more evenly can also reward irrelevant steps. Is the gain better credit assignment, or mostly reward smoothing? Also, these are shared team rewards, so this isn't directly solving credit assignment to individual agents.
  5. I'm not sure how much reward-design work has actually been saved. Someone still has to choose distance thresholds, duration requirements, and intermediate behaviors for the tests. Moving those choices into a tester can help, but the effort and sensitivity aren't measured. That's worth checking given the extra training time, especially the roughly doubled runtime on LBF.
Questions for authors
  1. How does a simple reward made from the same test indicators perform with a similar tuning budget? I'd include direct-indicator and incremental-progress versions.
  2. Why leave pass-fail outcomes out of the return model? What happens if the indicative inputs are similar or identical but the pass-fail rules prefer different trajectories?
  3. Can simpler smoothing or reward redistribution give the same benefit as the uniform prior? Do the learned rewards line up with the transitions that matter?
  4. What happens with badly chosen thresholds, missing requirements, or conflicting tests?
  5. When a test depends on history or action order, when can the single-transition reward model fit its return, and when does it fail?
3 comments
AuthorsOpenReview ↗

Clarifying the multi-agent extension and theoretical scope, with additional ablations.

Thank you for your thoughtful feedback. We have marked revisions in blue in the updated PDF.

W1/Q1: Do simple indicator rewards suffice?

Table 12 (p. 40). SMACv2 5v5 win (%) / surviving allies / eliminated enemies (time-averaged counts).

BackboneRewardProtossTerranZerg
QMIXDI0.0/3.2/0.05.0/3.1/0.48.0/2.9/1.4
IP66.0/3.8/1.981.0/4.2/1.627.0/3.1/2.1
Dense71.0/3.8/1.877.0/4.1/1.648.0/3.8/1.5
TdMARL72.0/4.0/1.869.0/4.1/1.843.0/3.8/1.8
MAPPODI0.0/3.2/0.01.0/3.1/0.35.0/2.9/1.1
IP47.0/3.4/1.866.0/3.9/1.634.0/3.5/1.8
Dense53.0/3.5/1.757.0/3.8/1.536.0/3.6/1.7
TdMARL55.0/3.7/1.756.0/3.8/1.627.0/3.5/1.8

Direct Indicator (DI) uses bt+αat+1+βkt+1b_t+\alpha a_{t+1}+\beta k_{t+1}; Incremental Progress (IP) uses rtdense+αΔat+βΔktr_t^{\mathrm{dense}}+\alpha\Delta a_t+\beta\Delta k_t. Here at,kta_t,k_t are surviving ally and cumulative eliminated enemy fractions, btb_t is sparse terminal reward, and Δxt=xt+1−xt\Delta x_t=x_{t+1}-x_t. TdMARL beats DI in all six settings. IP uses dense feedback and beats TdMARL on Terran (both backbones) and Zerg (MAPPO). Lines 499–507 (p. 10), 2031–2036 (p. 38). Figures 23–24 (p. 39).

W2: Why is the multi-agent extension nontrivial?

Joint successes can be scarce, evolving teammate policies shift data, and team returns underdetermine temporal allocation. Targeted sampling selects pairs differing on the least-passed test, preference tolerance limits sensitivity to indicator fluctuations, and the uniform prior constrains large, cancelling rewards. Table 1 (p. 10) supports all three. Lines 076–078 (p. 2).

W3: What does the bound guarantee?

Proposition 1 (p. 4) conditionally bounds test failure under separation and realizability. For environment return GeG_e, learned reward sum G^ϕ\hat G_\phi, and fixed calibration C(x)=ax+bC(x)=ax+b, a>0a>0, Eq. 7 (p. 4) and the triangle inequality give

Eπ∣C(G^ϕ)−Ge∣≤aϵπ+Eπ∣C(R^ψ)−Ge∣.\mathbb E_\pi|C(\hat G_\phi)-G_e|\le a\epsilon_\pi+\mathbb E_\pi|C(\hat R_\psi)-G_e|.

This separates decomposition error from environment-return alignment without those assumptions. Both terms are estimable from rollouts. Figure 4 (p. 9) reports 88.7–96.3% correct win–loss ordering. Lines 193–199 (p. 4).

W3/Q2: Why omit pass-fail inputs?

Indicative inputs express graded progress; pass-fail outcomes set preference priorities (lines 252–268, p. 5). Similar inputs can score differently; identical inputs cannot fit conflicting preferences. Indicators that determine passing avoid this ambiguity; unrestricted specifications need not. In CN, Td-MAPPO succeeds with Td-QMIX's tester, suggesting backbone dependence; QMIX's mixer may contribute. Lines 371–374 (p. 7), 1719–1727 (p. 32). Figures 13–14 (pp. 31–32).

W4/Q3: Does learned allocation beat smoothing?

Mean ± 95% CI across four seeds. Table 1 (p. 10).

BackboneAllocationCN coverageLBF pickupsProtoss win rate
QMIXUniform Split0.00±0.005.08±2.160.48±0.12
TdMARL0.03±0.018.92±1.200.72±0.03
MAPPOUniform Split0.00±0.008.88±0.500.07±0.13
TdMARL0.55±0.129.84±0.300.55±0.05

The soft prior stabilizes allocation while allowing variation. Uniform Split retains return learning and uses constant rt=R^(τ)/∑jdjr_t=\hat R(\tau)/\sum_j d_j to preserve returns under weights djd_j. TdMARL has higher means in all six comparisons, supporting learned allocation. Figures 6–7 (p. 24) link reward peaks to LBF pickups. Lines 483–485 (p. 9), 497–498 (p. 10), 2026–2029 (p. 38). Figure 22 (p. 37).

Shared team rewards follow cooperative MARL (Rashid et al., 2018; Yu et al., 2022). TdMARL allocates rewards across time; QMIX's factorization or MAPPO's centralized critic supports agent-level learning. Additional agent credit assignment, as in COMA (Foerster et al., 2018), is future work: line 539 (p. 10).

W5: What design work is saved?

SMACv2 tests specify victory, survival and elimination via terminal checks and trajectory averages (lines 1422–1443, p. 27). Dense rewards additionally require health/shield damage, bonuses, weights and normalization (Ellis et al., 2023; lines 1444–1456, p. 27). TdMARL learns step rewards, reducing manual shaping; tests transfer across backbones. LBF training takes roughly twice as long.

Q4: What if tests are flawed?

Poor thresholds can misdirect learning, like poor rewards. Unspecified requirements may go unmet. Conflicting tests induce trade-offs under the preference hierarchy (lines 252–266, p. 5). Mutually incompatible requirements cannot all be satisfied.

Q5: Can the model handle history?

A transition-only model cannot distinguish histories with identical inputs. A GRU or LSTM can condition rewards on event order if its state retains the relevant history. Lines 286–288 (p. 6).

Please do not hesitate to ask further questions. If we have addressed your concerns, please consider raising your confidence score or rating.

Thanks for the authors' response, my concerns are mainly addressed, I would like to increase my score.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank the reviewer for the positive feedback and for reconsidering the assessment. We are glad that our clarifications and additional experiments helped address the concerns. We greatly appreciate your constructive suggestions.

Reviewer_jPysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends test-driven RL (TdRL) to cooperative MARL. A designer writes pass-fail and indicative tests over joint trajectories; a hierarchical rule turns test results into preference labels; a return model is learned from them. Trajectory segment returns are factorized into per-step shared rewards that replace the environment reward in well-known baselines, such as QMIX and MAPPO. The method adds to TdRL the following: targeted pair sampling, a tolerance-based preference rule, and a soft uniform prior on the decomposition. A failure probability bound from the return decomposition error is analyzed, and MAPPO gradient equivalence is established. The method is tested on MPE, LBF, and SMAC environments, with sparse backbones and dense reward oracles.

Critical strengths

Td-QMIX/Td-MAPPO gain significant win rates on fully sparse SMACv2, according to Table 7. With results closely matching dense oracles, the presented method is shown to be effective.

The ablation study and sensitivity study across Tables 10-11 are extensive, and limitations are addressed candidly.

Critical weaknesses

While the paper offers a practical interface for specifying cooperative objectives, its weaknesses are twofold. The evaluation largely measures what the method itself optimizes, and the premise (tests are easier than rewards) and the theory are not substantiated.

Circular and mismatched evaluation. Nearly all main-text metrics (Tables 5, 6, 7, 9) are the tester’s own pass-fail or indicative outcomes: sustained coverage, safe separation, proximity, prey-contact frames, time-averaged allies/enemies and pairing. Fig. 4 plots the learned return against the same two quantities that are its inputs. Gains on these show the method follows its specification, not that coordination improved. The environment returns that could arbitrate are in the appendix (Figs. 8–9), where both QMIX-based methods stay low on Cooperative Navigation. Td-QMIX is below QMIX there on coverage (0.03 vs 0.11) and proximity (0.46 vs 0.65).

The “fully sparse” SMACv2 setting is also not what it seems. The sparse backbones and intrinsic-reward baselines see a ±1 terminal reward, while the tester computes time-averaged allies and enemies from global-state health at every step, which is effectively dense shaping. The dense oracle gives no penalty for ally damage or deaths (Eq. 52 discussion), so “Td outperforms dense on allied survival” holds by construction. The baseline that would show whether preference learning and decomposition add anything is missing: a hand-coded per-step reward from the same indicative quantities, or a uniform spread of the tester return. LBF (2 agents, 1 fruit) is already solved by the sparse backbones (about 9–10 pickups), so it cannot show a benefit. The auxiliary gains are 0.01–0.06 on the tester’s own metrics.

Unsubstantiated premise and thin theory. The testers are hand-set thresholds and scalarizations (LBF: distance ≤2.0 for ≥50% of steps, 1.5/10%, 1.2/10%). The MPE tests are explicitly modeled on the dense-reward objectives (Secs. 5.1–5.2), and the SMACv2 tester is comparable in size to the standard dense reward (damage, kills, win). There is no evidence of lower design effort or of robustness to mis-specified tests, which is the main risk of this paradigm. The three adaptations (targeted sampling, tolerance rule with skewness ordering, uniform prior) are ad hoc and not multi-agent-specific. The reward stays a single shared scalar, nothing exploits agent structure, and nothing is tested single-agent to support the non-stationarity argument. The prior pulls per-step rewards toward R/Z_γ, and the only evidence of learned temporal credit assignment is one LBF trajectory (Figs. 6–7).

Prop. 1 needs Δ-test-separation over all pairs and a factorized policy that succeeds almost surely. Neither is plausible on SMACv2: Fig. 4 reports 3.7–11.3% of win–loss pairs with equal or lower return, and the best dense oracle wins at most 0.77. Also, ε_π is defined under the evaluated policy’s distribution, while L_sum is trained on buffer data, which is exactly the non-stationarity gap the paper claims to address. Neither Δ nor ε_π is measured. Props. 2–3 are standard (policy gradient at the behaviour policy; 2η argmax error), and Prop. 3’s small-η assumption is where the experiments fail (Td-QMIX on Cooperative Navigation). So “establishing validity” is overstated, and the theory does not connect to what the experiments show. Please see questions below.

Questions for authors

Q1. Sec. 4.2 defines R̂ψ = fψ(z_ind), but preference rules (1)–(2) depend on z_pf. How does the return encode pass-fail separation (Def. 1)? Can you report the empirical Δ and ε_π, measured on the evaluated policy’s trajectories rather than buffer data, next to the observed failure probability? Is the Prop. 1 bound non-vacuous on SMACv2?

Q2. I assume QMIX/MAPPO on MPE are trained on the dense rewards (Eqs. 46, 48). If so, why does Td-MAPPO beat MAPPO on the dense return it never optimizes (Fig. 8), and was the MAPPO baseline tuned? Conversely, is Td-QMIX’s failure on Cooperative Navigation due to QMIX’s monotonic limitation or to the learned reward? What happens with a less restrictive mixer such as QPLEX?

Q3. Can you add (a) a hand-coded per-step reward built from the same indicative quantities (change in allies alive, enemies eliminated, plus terminal win) and (b) uniform spreading of the tester return (λ_step→∞)? Without these, it is unclear what preference learning and decomposition add over simply using the tests.

5 comments
AuthorsOpenReview ↗

Part 1: Evaluation and Reward Design

Thank you for your thoughtful feedback. We have marked revisions in blue in the updated PDF.

W1: Do the gains reflect task performance?

Section 5.2 reports SMACv2 win rates in Figure 2 (p. 7) and points to environment-return comparisons in Figures 8–10 (pp. 29–30). These returns use the same environment rewards for TdMARL and the baselines, while TdMARL trains on test-derived rewards.

Win rate measures battle success; allied survival and enemy elimination describe combat behavior. Sustained coverage and safe separation measure landmark coverage and collision avoidance, while prey contact measures interception. We prioritize these outcomes because they describe what the team achieves in practice. Their overlap with the tester is intentional: TdMARL starts from task requirements and learns rewards to meet them. Evaluating these behaviors helps avoid mistaking reward exploitation for task success. Figure 4 (p. 9) diagnoses return ordering, while task metrics and environment returns assess the resulting policies.

W1/Q2: Why do the MPE results depend on the backbone?

The MPE baselines use the dense rewards in Eqs. 46 and 48. MAPPO and Td-MAPPO share backbone settings and environment-step budgets, holding the backbone configuration fixed. Td-MAPPO's tests favor sustained coverage, which aligns with the dense objective. Its coverage increase from 0.23 to 0.55 is consistent with its higher dense return in Figure 8 (p. 29).

Both QMIX variants struggle with coverage. Figures 13–14 (pp. 31–32) show QMIX leaving one landmark uncovered and Td-QMIX failing to sustain stable assignments. Td-MAPPO's success with the same tester shows that these tests can support coordination. These observations suggest backbone dependence; the monotonic mixer is a possible contributor, without isolating it from reward-learning effects. QPLEX (Wang et al., 2021) would help test this interaction, but is not evaluated here. Lines 371–374 (p. 7), 1719–1727 (p. 32).

W2/W3: What does test design simplify?

“Sparse” refers to the terminal-only environment reward. The tester uses per-step state information, and Figure 5 (p. 16) explicitly illustrates the learned dense feedback. The contribution is that designers specify outcomes with simple checks, while TdMARL learns stepwise rewards. In SMACv2, terminal checks and trajectory averages summarize surviving allies and eliminated enemies. Dense reward design additionally specifies health/shield damage, bonuses and their weights. Lines 1413–1456 (p. 27).

The environments examine different questions. MPE aligns tests with dense objectives to assess comparable task outcomes. SMACv2 tests whether simple battle indicators yield useful dense feedback. LBF tests additional behavioral requirements when pickups are already near saturation. We therefore claim a reward-design paradigm centered on task requirements; further gains on individual LBF metrics are secondary once requirements are jointly met. Dense baselines still achieve higher win rates in several settings. Lines 311–320 (p. 6).

Tests still require appropriate thresholds and requirements. Poor or conflicting tests can misdirect learning, as poorly specified rewards can. The simplification concerns manual shaping choices; total design effort depends on the task.

W2/Q3: What do indicator-based controls show?

Table 12 (p. 40). SMACv2 5v5 win (%) / surviving allies / eliminated enemies (time-averaged counts).

BackboneRewardProtossTerranZerg
QMIXDI0.0/3.2/0.05.0/3.1/0.48.0/2.9/1.4
IP66.0/3.8/1.981.0/4.2/1.627.0/3.1/2.1
Dense71.0/3.8/1.877.0/4.1/1.648.0/3.8/1.5
TdMARL72.0/4.0/1.869.0/4.1/1.843.0/3.8/1.8
MAPPODI0.0/3.2/0.01.0/3.1/0.35.0/2.9/1.1
IP47.0/3.4/1.866.0/3.9/1.634.0/3.5/1.8
Dense53.0/3.5/1.757.0/3.8/1.536.0/3.6/1.7
TdMARL55.0/3.7/1.756.0/3.8/1.627.0/3.5/1.8

Let at,kta_t,k_t be surviving ally and cumulative eliminated enemy fractions, btb_t the sparse terminal reward, and Δxt=xt+1−xt\Delta x_t=x_{t+1}-x_t. Direct Indicator (DI) uses bt+αat+1+βkt+1b_t+\alpha a_{t+1}+\beta k_{t+1}. Incremental Progress (IP) uses rtdense+αΔat+βΔktr_t^{\mathrm{dense}}+\alpha\Delta a_t+\beta\Delta k_t. Eq. 52 (p. 27) uses the official dense reward (Ellis et al., 2023). Higher survival alone reflects the tester's objective; these controls assess what learning adds when the same indicators are available.

TdMARL beats DI in all six settings, with win rates of 27–72% versus 0–8%. TdMARL and IP each lead in three settings. IP builds on engineered dense feedback and requires additional weighted progress terms. These results support learning useful rewards from task tests and competitive performance with fewer manual shaping choices. Lines 499–507 (p. 10), 2031–2036 (p. 38). Figures 23–24 (p. 39).

AuthorsOpenReview ↗

Part 2: Learning Mechanisms and Theoretical Scope

W3: Why shared rewards and these adaptations?

TdMARL follows fully cooperative MARL, where agents optimize a shared team reward (Foerster et al., 2018; Rashid et al., 2018; Yu et al., 2022). It learns one scalar reward per step, shared by all agents; the backbone handles agent-level policy learning. Under partial observability, changing teammate policies can make each agent's effective environment non-stationary (Foerster et al., 2017; Lowe et al., 2017).

These conditions motivate the adaptations. When joint successes are scarce, targeted sampling selects pairs differing on the least-passed test. As policies evolve, preference tolerance limits sensitivity to small indicator fluctuations. Because team returns admit many temporal allocations, the soft uniform prior discourages large, cancelling rewards in the shared training signal. These mechanisms stabilize cooperative learning; Table 1 (p. 10) supports their use with both backbones. Lines 076–078 (p. 2).

W3/Q3: Does learned allocation add value beyond smoothing?

Table 1 (p. 10). Mean ± 95% CI across four seeds.

BackboneAllocationCN coverageLBF pickupsProtoss win rate
QMIXUniform Split0.00±0.005.08±2.160.48±0.12
TdMARL0.03±0.018.92±1.200.72±0.03
MAPPOUniform Split0.00±0.008.88±0.500.07±0.13
TdMARL0.55±0.129.84±0.300.55±0.05

The prior provides a soft reference, allowing rewards to differ across steps. Uniform Split instead retains return learning and assigns rt=R^(τ)/∑jdjr_t=\hat R(\tau)/\sum_j d_j to each valid step, preserving the predicted return under the return-matching weights djd_j.

Full TdMARL has higher means in all six comparisons. MAPPO's Protoss win rate rises from 0.07 to 0.55, and QMIX's LBF pickups increase from 5.08 to 8.92. Together with the prior-removal ablation, these results support regularizing allocation while retaining learned temporal differences. Figures 6–7 (p. 24) provide complementary behavioral evidence from one LBF rollout. Lines 483–485 (p. 9), 497–498 (p. 10), 2026–2029 (p. 38). Figure 22 (p. 37).

W4: How does the theory connect to the experiments?

Proposition 1 (p. 4) bounds test failure under separation and policy realizability. These are conditional guarantees; the empirical ranking results do not establish their assumptions. We therefore complement the bound with measurable return errors. Let GenvG_{\mathrm{env}} denote environment return, G^ϕ\hat G_\phi the discounted learned reward sum, and c(x)=ax+bc(x)=ax+b, a>0a>0, a fixed calibration. Eq. 7 (p. 4) and the triangle inequality give

Eπ∣c(G^ϕ)−Genv∣≤aϵπ+Eπ∣c(R^ψ)−Genv∣.\mathbb E_\pi|c(\hat G_\phi)-G_{\mathrm{env}}|\le a\epsilon_\pi+\mathbb E_\pi|c(\hat R_\psi)-G_{\mathrm{env}}|.

This separates decomposition error from environment-return alignment without requiring separation or realizability. Both terms can be estimated on the evaluated policy's rollouts. The buffer loss trains the reward model; it does not by itself establish a small error under a different policy distribution. Lines 193–199 (p. 4).

Propositions 2–3 (pp. 4, 20) apply standard results to MAPPO's gradient at the behavior policy and QMIX's value-approximation error. They clarify conditional compatibility with these backbones. CN performance alone cannot isolate mixer error from reward-learning and optimization errors.

Q1: How are pass-fail priorities learned and evaluated?

Table 9 (p. 35). Fresh-rollout gap and decomposition error; pairwise error from Figure 4 (p. 9).

SettingGap Δ^\hat\DeltaError ϵ^π\hat\epsilon_\piPairwise error
Protoss Td-QMIX−2.2620.2363.69%
Protoss Td-MAPPO−2.1680.2776.21%
Terran Td-MAPPO−2.1740.32111.30%
Zerg Td-QMIX−2.4660.3504.97%

Pass-fail outcomes set preference priorities through the training labels, while indicative inputs provide graded progress. Identical inputs necessarily receive identical scores, so conflicting preferences at identical inputs cannot be fitted by this return model.

With policies and learned models frozen, we collect 256 fresh rollouts per seed, or 1,024 per setting. The gap is the minimum passing score minus the maximum failing score; decomposition error is the mean absolute difference between the discounted learned reward sum and predicted return. Negative gaps capture worst-case reversals, while Figure 4 reports correct ordering in 88.7–96.3% of sampled pairs. Errors of 0.236–0.350 quantify reconstruction on the standardized return scale. These measurements support empirical alignment across return learning and decomposition. Proposition 1 characterizes the stronger positive-margin regime and does not give a numerical certificate for these runs. Lines 443–448 (p. 9), 1816–1831 (p. 34).

Please do not hesitate to ask further questions. If we have addressed your concerns, please consider raising your rating.

AuthorsOpenReview ↗

Follow-up on Rebuttal Feedback

Dear Reviewer,

Hope you are doing well. We have submitted our rebuttal and would greatly appreciate it if you could share any additional feedback when you are available. Thank you for your valuable time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Revisions

Dear Area Chair, Program Chairs, and Reviewers,

Thank you for your thoughtful reviews. Reviewer qdjc highlighted the idea behind TdMARL: designers can describe what a team should achieve and let the algorithm learn how to reward progress toward it. Reviewer FioE recognized the value of testing both on-policy and off-policy backbones. Reviewer jPys noted the SMACv2 win-rate gains and asked what reward learning adds beyond access to the task indicators.

In response, we made four revisions.

  • Reward learning from task indicators. Across three SMACv2 scenarios and both backbones, TdMARL achieved higher win rates than directly rewarding the same survival and elimination indicators in all six comparisons (27–72% versus 0–8%). Against a stronger control that adds weighted changes in these indicators to the official dense reward, each method led in three settings (Table 12, Figures 23–24).
  • Temporal reward allocation. We added Uniform Split, which distributes each predicted return evenly across steps. Learned allocation achieved higher mean performance in all six comparisons across Cooperative Navigation, LBF, and Protoss 5v5. On Protoss, MAPPO’s mean win rate rose from 7% to 55% (Table 1, Figure 22).
  • Theoretical scope and reward reliability. Section 3 clarifies the assumptions behind Proposition 1 and separates decomposition error from alignment with environment return. On 1,024 fresh rollouts per setting, mean absolute decomposition errors ranged from 0.236 to 0.350 on the standardized return scale. We observed some pass–fail ordering reversals in all four settings, alongside 88.7–96.3% correct win–loss ordering in Figure 4.
  • Behavioral differences across backbones. With the same Cooperative Navigation tester, Td-MAPPO achieved 55% sustained coverage, while Td-QMIX achieved 3%. Both QMIX variants struggled to maintain landmark assignments. One possible reason is that QMIX’s monotonic mixer makes credit assignment difficult when the value of an agent’s landmark choice depends on which landmarks its teammates cover (Section 5.2, Figures 12–14).

The revised PDF includes the new experiments, tables, learning curves, and analyses, with revisions marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.