- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the identifiability of per-agent return distributions in distributional value decomposition. The authors show that team-only training does not identify individual return distributions and that matched-quantile summation imposes comonotone recomposition. They propose a decoupled supervised critic and evaluate it on a heterogeneous search task and Gigastep.
Critical strengths
1.The paper identifies two distinct structural issues in distributional value decomposition—non-identification of per-agent return distributions and inconsistency induced by matched-quantile recomposition—and supports them with both theoretical analysis and oracle experiments.
2.The empirical evaluation is well aligned with the claims, particularly through the credit-split intervention, calibration analysis, and downstream per-agent risk decisions.
Critical weaknesses
1.The proposed decoupled critic resolves the recomposition conflict by removing the coupling between team and per-agent predictions, but consequently gives up consistency between the two distributions. This limits it as a general solution to distributional value decomposition.
2.The predictive gains attributed to decoupling are not fully isolated from differences in conditioning information: the separate team head observes the full team observation, and the paper’s own team-input control substantially narrows the advantage over the supervised matched-sum critic.
Questions for authors
1.Could the authors clarify how much of the predictive improvement should be attributed specifically to decoupling, given that the team-input matched-sum control substantially narrows the CRPS gap?
2.Could the authors clarify how the remaining-horizon bias in the search task affects the interpretation of the downstream per-agent risk-decision results?
2 comments
Post-rebuttal assessment
Thank you for the detailed response and the additional experiments. The rebuttal substantially addresses my main concerns.
- Input-matched controls and the benefit of decoupling (W2 & Q1): The new input- and capacity-matched comparison in Table 1 clarifies the attribution of the gains. Per-agent supervision accounts for most of the individual prediction accuracy, while the decoupled architecture improves calibration at both levels and lowers team CRPS. This gives a clearer account of its contribution.
- Remaining-horizon bias (Q2): Adding the remaining episode fraction to the critic and evaluating decisions at training-distribution states directly addresses my concern about horizon shift. The additional results strengthen the evidence for the reported decision-making benefit.
- Consistency and calibration (W1): The response clarifies why summing per-agent quantiles at matched levels imposes a comonotonicity assumption that can distort calibration. The separate team head avoids this constraint and improves calibration at both levels. The reported average signed mean discrepancy is below 1% of the mean team return, and the new PPO experiments provide further evidence of practical value through reduced collision rates.
Consistency between the separately predicted distributions remains a limitation, but the revised experiments clarify the benefits and scope of the design. Together with the theoretical analysis of non-identification and matched-quantile recomposition, these results support my positive assessment. I maintain my rating of 3: Weak acceptance.
Response to Reviewer QS8b
We thank the reviewer for noting that the paper supports both structural issues with theory and oracle experiments. The main concern is how much of the improvement is due to decoupling; both weaknesses and both questions led to new experiments. Numbers are for the search task, critic seeds 4 to 9, at training-distribution states.
Weakness 1: the separate team head trades exact additivity for calibration at both levels (Section 4).
By Proposition 3 the matched sum is the quantile function of the comonotone sum of the returns, so enforcing additivity imposes a consistency the returns do not have in general. At equal inputs and size the supervised matched sum over-covered the team return (0.972) and under-covered the per-agent returns (0.787 to 0.795; nominal 0.906), while our critic had 0.906 to 0.913 and 0.875 to 0.880. Without this constraint the two heads of our critic had a small mean discrepancy: the signed discrepancy between the summed per-agent means and the team prediction, averaged over states, was below 1% of the mean team return. The two heads also share the per-agent trunk, which the team head pools.
Weakness 2 and Q1: decoupling gives calibration at both levels and a lower team CRPS (Section 5.4, Table 1).
Fully input-matched comparison (every per-agent head reads its own and the team observation; 366.5k against 366.9k parameters), policy 1 / policy 2:
Most of the per-agent accuracy is due to per-agent supervision, as the reviewer suggested; decoupling adds calibration at both levels and a lower team CRPS. It also changed the learned policies (Section 5.5): with PPO on fifteen seeds fixed before their runs, our critic gave 2.00 collisions per episode against 2.41 (lower in 12 of 15 pairs; confirmatory test p = 0.008) and 2.00 against 2.56 for a matched sum of equal size (9 of 10 pairs, p = 0.002).
Q2: with the time input, the decision advantage extends to training-distribution states (Section 5.4, Appendix M).
Without a time input a critic cannot distinguish identical observations with different remaining horizons. The critic now receives the remaining episode fraction; the actor does not, so the frozen policies act identically. With this input the observation-head variant of our critic reduced its per-agent CRPS at policy 1 from 4.69 to 2.92; its held-out error was 0.316 and 0.333 against 0.471 and 0.473 for team-only at policy 1 and 0.347 and 0.340 against 0.423 and 0.400 at policy 2 (0.318 and 0.333 at policy 1 with the pooled team head); all four intervals of the difference exclude zero.
We hope these results address your concerns and are happy to answer further questions.