acceptodds
Under review as a conference paper at ICLR 2027

Per-agent return distributions in value decomposition

Abstract

Distributional value decomposition for cooperative multi-agent reinforcement learning predicts a return distribution for each agent and combines the per-agent quantiles into a team distribution and risk-sensitive variants act on the per-agent distributions. We show that training on the team return does not identify them: at every state many decompositions give the same team quantiles and any decision rule that uses only team-level data violates a per-agent risk constraint with probability at least one half in some environment. Supervising each agent with its own return removes this ambiguity, but summing quantiles at matched levels, as DFAC-style mixers do, is comonotone recomposition, which for finite-variance returns overstates the variance of the team return unless agent returns are comonotone. We propose a critic that trains per-agent quantile heads on individual returns and predicts the team distribution with a separate head. In a heterogeneous search task and in Gigastep, team-only training left per-agent coverage far from nominal in most settings and dependent on the policy and on how credit was shared. The proposed critic lowered per-agent CRPS in every setting, kept team coverage at the team-only level where supervised matched-sum critics raised it and improved per-agent risk decisions in the search task.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, value factorization, distributional reinforcement learning, risk-sensitive reinforcement learning, calibration, cooperative multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QS8bOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the identifiability of per-agent return distributions in distributional value decomposition. The authors show that team-only training does not identify individual return distributions and that matched-quantile summation imposes comonotone recomposition. They propose a decoupled supervised critic and evaluate it on a heterogeneous search task and Gigastep.

Critical strengths

1.The paper identifies two distinct structural issues in distributional value decomposition—non-identification of per-agent return distributions and inconsistency induced by matched-quantile recomposition—and supports them with both theoretical analysis and oracle experiments.

2.The empirical evaluation is well aligned with the claims, particularly through the credit-split intervention, calibration analysis, and downstream per-agent risk decisions.

Critical weaknesses

1.The proposed decoupled critic resolves the recomposition conflict by removing the coupling between team and per-agent predictions, but consequently gives up consistency between the two distributions. This limits it as a general solution to distributional value decomposition.

2.The predictive gains attributed to decoupling are not fully isolated from differences in conditioning information: the separate team head observes the full team observation, and the paper’s own team-input control substantially narrows the advantage over the supervised matched-sum critic.

Questions for authors

1.Could the authors clarify how much of the predictive improvement should be attributed specifically to decoupling, given that the team-input matched-sum control substantially narrows the CRPS gap?

2.Could the authors clarify how the remaining-horizon bias in the search task affects the interpretation of the downstream per-agent risk-decision results?

2 comments
AuthorsOpenReview ↗

Response to Reviewer QS8b

We thank the reviewer for noting that the paper supports both structural issues with theory and oracle experiments. The main concern is how much of the improvement is due to decoupling; both weaknesses and both questions led to new experiments. Numbers are for the search task, critic seeds 4 to 9, at training-distribution states.

Weakness 1: the separate team head trades exact additivity for calibration at both levels (Section 4).

By Proposition 3 the matched sum is the quantile function of the comonotone sum of the returns, so enforcing additivity imposes a consistency the returns do not have in general. At equal inputs and size the supervised matched sum over-covered the team return (0.972) and under-covered the per-agent returns (0.787 to 0.795; nominal 0.906), while our critic had 0.906 to 0.913 and 0.875 to 0.880. Without this constraint the two heads of our critic had a small mean discrepancy: the signed discrepancy between the summed per-agent means and the team prediction, averaged over states, was below 1% of the mean team return. The two heads also share the per-agent trunk, which the team head pools.

Weakness 2 and Q1: decoupling gives calibration at both levels and a lower team CRPS (Section 5.4, Table 1).

Fully input-matched comparison (every per-agent head reads its own and the team observation; 366.5k against 366.9k parameters), policy 1 / policy 2:

MeasureOursSupervised matched sum
Per-agent coverage (nominal 0.906)0.875 / 0.8800.787 / 0.795
Team coverage0.906 / 0.9130.972 / 0.972
Per-agent CRPS2.802 / 3.0162.818 / 3.045
Team CRPS7.47 / 8.628.02 / 9.21
Held-out flagging error0.302 to 0.3090.312 to 0.321
Within-state rank correlation0.593 / 0.5540.618 / 0.575

Most of the per-agent accuracy is due to per-agent supervision, as the reviewer suggested; decoupling adds calibration at both levels and a lower team CRPS. It also changed the learned policies (Section 5.5): with PPO on fifteen seeds fixed before their runs, our critic gave 2.00 collisions per episode against 2.41 (lower in 12 of 15 pairs; confirmatory test p = 0.008) and 2.00 against 2.56 for a matched sum of equal size (9 of 10 pairs, p = 0.002).

Q2: with the time input, the decision advantage extends to training-distribution states (Section 5.4, Appendix M).

Without a time input a critic cannot distinguish identical observations with different remaining horizons. The critic now receives the remaining episode fraction; the actor does not, so the frozen policies act identically. With this input the observation-head variant of our critic reduced its per-agent CRPS at policy 1 from 4.69 to 2.92; its held-out error was 0.316 and 0.333 against 0.471 and 0.473 for team-only at policy 1 and 0.347 and 0.340 against 0.423 and 0.400 at policy 2 (0.318 and 0.333 at policy 1 with the pooled team head); all four intervals of the difference exclude zero.

We hope these results address your concerns and are happy to answer further questions.

Reviewer_QS8bOpenReview ↗

Post-rebuttal assessment

Thank you for the detailed response and the additional experiments. The rebuttal substantially addresses my main concerns.

  • Input-matched controls and the benefit of decoupling (W2 & Q1): The new input- and capacity-matched comparison in Table 1 clarifies the attribution of the gains. Per-agent supervision accounts for most of the individual prediction accuracy, while the decoupled architecture improves calibration at both levels and lowers team CRPS. This gives a clearer account of its contribution.
  • Remaining-horizon bias (Q2): Adding the remaining episode fraction to the critic and evaluating decisions at training-distribution states directly addresses my concern about horizon shift. The additional results strengthen the evidence for the reported decision-making benefit.
  • Consistency and calibration (W1): The response clarifies why summing per-agent quantiles at matched levels imposes a comonotonicity assumption that can distort calibration. The separate team head avoids this constraint and improves calibration at both levels. The reported average signed mean discrepancy is below 1% of the mean team return, and the new PPO experiments provide further evidence of practical value through reduced collision rates.

Consistency between the separately predicted distributions remains a limitation, but the revised experiments clarify the benefits and scope of the design. Together with the theoretical analysis of non-identification and matched-quantile recomposition, these results support my positive assessment. I maintain my rating of 3: Weak acceptance.

Reviewer_ZPQqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work propose a critic that trains per-agent quantile heads on individual returns and predicts the team distribution with a separate head. In a heterogeneous search task and in Gigastep, team-only training left per-agent coverage far from nominal in most settings and dependent on the policy and on how credit was shared.

Critical strengths
  1. The paper identifies a genuine and under-appreciated identification problem in distributional MARL.
  2. The decoupled supervised critic is a clean, minimal design, and the empirical protocol is unusually careful.
Critical weaknesses
  1. The practical scope is limited by the requirement of individual rewards, and the paper does not fully address how restrictive this is.
  2. The experimental comparison is restricted to critic training with frozen policies, and the paper’s claims about policy learning are not tested.
Questions for authors
  1. The method requires observable per-agent rewards during centralized training. Please add a table or paragraph comparing the environments used in the paper with common cooperative MARL benchmarks along the axes of reward observability, team size, and whether per-agent returns are meaningful. Discuss whether difference rewards, counterfactual baselines, or Shapley-based credit could provide per-agent targets when individual rewards are unavailable, and whether the decoupled supervised critic could be combined with those credit-assignment methods. This would clarify whether the contribution is a general critic design or a diagnostic for the subset of environments with individual rewards.
  2. The current experiments freeze the policy and train critics from scratch. This is a clean diagnostic, but the paper’s framing (“training a critic,” “risk-sensitive decisions”) could be read as implying downstream policy benefits. Please either (i) add an experiment where the decoupled supervised critic is used inside PPO to train a risk-sensitive or risk-neutral policy, and compare team return and per-agent CVaR against matched-sum critics; or (ii) explicitly state in the abstract and introduction that the contribution is a critic-level identification and calibration result, and that policy-learning effects are left to future work.
1 comment
AuthorsOpenReview ↗

Response to Reviewer ZPQq

We thank the reviewer for describing the identification problem as genuine and the protocol as careful. The main concerns are the effect on policy learning and the need for per-agent rewards; both led to new experiments.

Q3 and weakness 2: policies trained with our critic collided less and its per-agent quantiles raised the worst-agent CVaR (Section 5.5, Appendix O, Figure 11).

We trained policies from random initialisation with PPO in the search task with our critic and with the strongest baseline, the matched-sum critic trained with the same per-agent targets; the actor and its update are identical in both arms. Seeds 0 to 5 and fifteen further seeds fixed before their runs:

Measure (ours against matched sum)Seeds 0 to 5Fifteen further seeds
Team return313.6 against 312.0310.7 against 293.9
Collisions per episode1.89 against 2.272.00 against 2.41 (lower in 12 of 15)
Critic coverage closer to nominal, both heads6 of 6 pairs15 of 15 pairs

Collisions were fixed as the outcome before the further seeds (confirmatory test over ten pairs, exact one-sided p = 0.008). A matched sum of equal critic size (127,796 against 128,192 parameters) also collided more: 2.56 against 2.00 (9 of 10 pairs, p = 0.002).

The per-agent quantiles also raised the worst-agent CVaR (Section 5.5, Table 23). A risk gate applies a CVaR term to the agent-steps that the critic ranks lowest (gate level and weight fixed before the runs). With our critic it raised the worst-agent CVaR from -3.11 to -0.76 (higher in 9 of 10 seeds, p = 0.002) at an unchanged team return (309.8 against 309.7); with a critic trained on the team return alone, the gated policy reached -2.70 (ours ahead in 9 of 10 seeds, p = 0.004). Against the gated matched sum, ours had a higher team return (309.8 against 291.8), fewer collisions (1.81 against 2.02) and coverage closer to nominal at both levels (10 of 10 seeds), while the gated matched sum reached the higher worst-agent CVaR (0.13 against -0.76). The introduction and Section 5.5 now report these policy results.

Q1 and weakness 1: the method also applies to team-reward benchmarks (Appendix L, Table 20; Section 5.6; Appendix P).

Of nine cooperative benchmarks, five return per-agent rewards or compute per-agent terms before summing them (particle environments, Overcooked, Level-Based Foraging, multi-robot warehouse, Gigastep); four define the reward for the team (SMAC, SMACv2, Multi-Agent MuJoCo, Google Research Football). The requirement applies to centralised training in simulation only.

For a team reward, the target of an agent is the team reward minus the team reward with that agent's torques at zero, one extra simulator step per agent. Brax Ant (four-leg partition of Multi-Agent MuJoCo, one team reward), critics under one fixed walking policy (five seeds, three for team-only; nominal 0.906):

CriticTeam coverageTeam CRPSPer-agent coverage
Ours0.9141.070.907
Matched sum0.17310.810.359
Team only0.9061.130.045

These targets do not sum to the team return, so a matched sum cannot fit both levels: its combined mean error was 19.1 against 0.04 for ours (5 of 5 seeds). The values use a Huber threshold of 0.003; Table 24 lists every threshold and a matched sum of equal size.

Q2: difference rewards, counterfactual baselines and Shapley credit give per-agent targets (Section 4, Appendix P).

Yes. Each gives every per-agent head a target of its own, which is the condition of our identification result; the separate team head fits these targets where a matched sum cannot. In the search task with difference-reward targets (six critic seeds), our critic had per-agent coverage 0.876 and team coverage 0.907 against 0.781 and 0.968 for the matched sum and team CRPS 7.60 against 8.86, each better in 6 of 6 seeds (p = 0.016; Table 26).

We hope these results address your concerns and are happy to answer further questions.

Reviewer_odc4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that fitting a team return distribution does not identify the individual return distributions used to construct it. It also analyzes the dependence assumption behind summing matched quantiles. Experiments with frozen policies in Search and Gigastep compare individual supervision and a separate team head using calibration, CRPS, and risk-based decisions.

Critical strengths
  1. A clear warning about interpreting value heads. The counterexamples establish that a correct team prediction can coexist with incorrect individual risks. This matters when utility heads are used for per-agent risk constraints rather than only for action selection (Section 3).

  2. Experiments that test the proposed explanation. Oracle recomposition removes network-fitting error, while the credit-split sweep changes individual returns without changing the policy or team reward. Together they provide direct evidence for the dependence problem (Section 5; Appendices F–H).

  3. Useful controls and qualifications. The capacity/input controls and same-commit Search replication strengthen the results. The paper also reports that the predicted increase in across-state decision benefits does not appear in the credit-split sweep (Appendices G–I).

Critical weaknesses
  1. The added benefit of decoupling is narrower than that of supervision. In the team-input control, the supervised matched sum has lower individual CRPS and a team-CRPS difference whose interval includes zero, although decoupling retains better coverage. Its per-agent heads now receive more information than the decoupled heads, so this is not a fully input-matched comparison. Tables 17 and 18 also show little consistent decision benefit over the supervised matched sum.

  2. The decision experiment does not validate a fixed risk budget. Each critic has a threshold fitted to the oracle feasible fraction. Search also trains across remaining horizons but tests decisions at initial states; Appendix E acknowledges the resulting shift. These are useful ranking diagnostics, but do not show that the predicted CVaR can directly enforce a specified risk constraint.

  3. The empirical scope is narrower than the theoretical scope. The non-identification argument covers both matched-sum and mean-shape mixers, but the critic experiments test matched sums under frozen policies only. As Section 7 acknowledges, the observed calibration effects should not be read as measured failures of all distributional MARL methods or as evidence about policy learning.

Questions for authors
  1. Is an input-matched comparison available in which both architectures give every per-agent head the team observation? This would clarify the team-CRPS benefit attributable specifically to decoupling.

  2. Using the existing decision predictions, how do methods compare at the same numerical CVaR budget without fitting critic-specific thresholds?

  3. Is there evidence that the Search decision advantage survives the horizon shift discussed in Appendix E? If not, please keep the practical claim limited to the reported ranking test.

1 comment
AuthorsOpenReview ↗

Response to Reviewer odc4

We thank the reviewer for noting that a correct team prediction can coexist with incorrect individual risks; each of the three questions led to a new experiment. The main concern is how much of the gain is due to decoupling at equal inputs. Numbers are for the search task at training-distribution states unless stated.

Q1 and weakness 1: at equal inputs and size, our critic was closer to nominal at both levels (Section 5.4, Table 1).

In both architectures every per-agent head now reads its own and the team observation, at equal size. The comparison led to the refined critic, whose team head reads the mean of the per-agent trunk features, so the team loss also trains the per-agent trunk; both architectures also receive the remaining episode fraction (Q3). Refined critic against the supervised matched sum (366.5k against 366.9k parameters, critic seeds 4 to 9), policy 1 / policy 2:

MeasureOursSupervised matched sum
Per-agent coverage (nominal 0.906)0.875 / 0.8800.787 / 0.795
Team coverage0.906 / 0.9130.972 / 0.972
Per-agent CRPS2.802 / 3.0162.818 / 3.045
Team CRPS7.47 / 8.628.02 / 9.21
Held-out flagging error0.302 to 0.3090.312 to 0.321
Within-state rank correlation0.593 / 0.5540.618 / 0.575

Per-agent CRPS was lower for ours in 5 of 6 seeds at each policy. The sum constraint widens the team distribution (coverage 0.97) while the per-agent intervals remain too narrow (0.79), as Proposition 3 predicts. We agree that per-agent supervision produces most of the per-agent accuracy; the introduction and Appendix M state it.

Q2 and weakness 2: one numerical budget for every critic, after one calibration map per critic (Appendix N).

The budget is the same number for every critic and no threshold is fitted per budget. Each critic receives one monotone map, the quantile mapping of its worst-agent CVaR on 512 calibration states, fitted once and tested on 512 other states. At the median budget the error was 0.319 and 0.331 for ours against 0.470 and 0.474 for the team-only critic (two state samples, policy 1) and 0.301 to 0.309 against 0.311 to 0.322 for the matched sum with the team input; at the 75th-percentile budget the matched sum was lower (0.183 to 0.225 against 0.221 to 0.261).

Q3: with the time input, the decision advantage extends to training-distribution states (Section 5.4, Appendix M).

The critic now receives the remaining episode fraction; the actor does not, so the frozen policies act identically. The observation-head variant of our critic reduced its per-agent CRPS at policy 1 from 4.69 to 2.92; its held-out error was 0.316 and 0.333 against 0.471 and 0.473 for the team-only critic at policy 1 and 0.347 and 0.340 against 0.423 and 0.400 at policy 2 (0.318 and 0.333 at policy 1 with the pooled team head). All four intervals of the difference exclude zero.

Weakness 3: the critic also changed the learned policies (Section 5.5, Appendix O, Figure 11).

With PPO from random initialisation and the same actor in both arms, our critic was closer to nominal on both heads in each of the 21 seed pairs. Its policies collided less: 1.89 against 2.27 per episode on seeds 0 to 5 and 2.00 against 2.41 on fifteen seeds fixed before their runs (confirmatory p = 0.008), at a higher team return (310.7 against 293.9); a matched sum of equal size also collided more (2.56 against 2.00, p = 0.002). A risk gate on the per-agent quantiles raised the worst-agent CVaR from -3.11 to -0.76 (9 of 10 seeds, p = 0.002) at an unchanged team return; with a critic trained on the team return alone, the gated policy reached -2.70.

We hope these results address your concerns and are happy to answer further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers. The reviews raised three questions: whether the comparison with the baselines is fair, whether the critic matters beyond prediction accuracy and whether the method applies beyond the search task. Each led to new experiments and two led to refinements of the critic; changes in the revised PDF are marked in blue.

Contribution. The paper proposes a critic that supervises each agent with its own return and predicts the team distribution with a separate head, instead of summing per-agent quantiles. By Proposition 3 the matched sum is the comonotone sum of the returns, so exact additivity imposes a dependence that the returns do not have in general: at equal inputs and size, the supervised matched sum over-covered the team return (0.972) and under-covered the per-agent returns (0.787 to 0.795, nominal 0.906), while the separate head was calibrated at both levels.

Refinements of the critic. (1) Every head receives the remaining episode fraction. (2) The team head reads the mean of the per-agent trunk features, so the team loss also trains the per-agent trunk.

Main outcomes.

  1. Downstream benefit. In PPO from random initialisation, policies trained with the proposed critic collided less (2.00 against 2.41 collisions per episode on fifteen seeds fixed before their runs, confirmatory p = 0.008) at a higher team return (310.7 against 293.9); they also collided less than with a matched-sum critic of equal size (2.00 against 2.56). A risk gate on the per-agent quantiles raised the worst-agent CVaR from -3.11 to -0.76 at an unchanged team return.
  2. Fair comparison. At equal inputs and size, per-agent supervision accounts for most of the per-agent accuracy; decoupling adds calibration at both levels and a lower team CRPS.
  3. Generality. Five of nine cooperative benchmarks return per-agent rewards or compute per-agent terms; for the other four a per-agent target is computed with one counterfactual simulator step. On Brax Ant (one team reward, Huber threshold 0.003) the critic covered team and per-agent returns at 0.914 and 0.907 against 0.173 and 0.359 for the matched sum.
ConcernReviewersNew evidenceLocation
Fair comparison at equal inputs and sizeodc4 Q1, QS8b W2, Q1input-matched critics of equal sizeSec. 5.4, Table 1
One numerical CVaR budgetodc4 Q2, W2one calibration map per critic; lower error than both baselines at the median budget, matched sum with team input lower at the 75th percentileApp. N
Remaining-horizon shiftodc4 Q3, QS8b Q2time input; lower decision error than team-only in 4 of 4 comparisonsSec. 5.4, App. M
Effect on learned policiesodc4 W3, ZPQq Q3, W2PPO over 21 seed pairs, matched sum of equal size, risk gateSec. 5.5, App. O
Benchmarks with per-agent rewardsZPQq Q1, W1nine-benchmark table; Brax AntApp. L, Sec. 5.6, App. P
Other credit targetsZPQq Q2difference-reward targets in the search taskSec. 4, App. P
Consistency between the headsQS8b W1signed mean discrepancy below 1% of the mean team returnSec. 4

Presentation. Figure 1 is new; Figure 3 shows the two critics side by side and Appendix C the training loop. Section 7 states the limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.