- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why dense cooperative shaping can collapse performance in MARL. The claim is that reward-scale miscalibration moderately lowers the acquisition rate of each necessary event, and that tasks requiring all events to succeed multiply these losses ("conjunctive amplification"). Evidence comes from a Partial Relay grid (rho x lambda, 225 runs), an additive-versus-conjunctive switch task (joint success tracks the product of marginals, r = 0.998), and relay tasks where running reward normalization recovers most of the loss. Overcooked, SMACv2, PCGrad and static-rescaling controls are also reported. An Event Order Consistency (EOC) probe is proposed to decide when to calibrate.
Critical strengths
The controls are careful: matched-magnitude noise, dose and exposure sweeps, order-free variants, per-stream normalization, and frozen versus running normalizers.
The practical point is useful. A good fixed reward scale exists but can only be identified in hindsight, so running normalization matters in cooperative MARL.
Critical weaknesses
The mechanism is arithmetic, not learning. If success requires all L roughly independent events, success equals the product of the marginals by definition, and r = 0.998 over configurations ranging from 0 to 1 is guaranteed by range. The paper's own switch task shows per-event degradation is the same under additive and conjunctive reward (p > 0.07). So conjunctive structure changes the metric, not what the agents learn. Calling this "the missing link" and "the causal mechanism" is an overclaim.
"Scale, not content" is not cleanly shown. (a) The calibration subtracts a running mean, which adds a per-step bonus or cost and is not policy-invariant in episodic tasks. RMS-only scaling gives 0.686 versus 0.839 with centering. (b) Advantages are batch-normalized in every run, so the policy gradient is already scale-invariant. The effect must enter through the critic, and the paper never shows this. (c) After normalization, cooperative content still hurts in most tasks: normalized minimal beats normalized full on Hidden Gift (0.973 vs 0.845) and Partial Relay at rho = 1 (0.98 vs 0.84). Only Sequential Plate benefits.
The Overcooked baselines (0.42 vs 1.78 deliveries) are far below standard PPO results, so "fourfold" compares undertrained agents. SMACv2 is not a raw-versus-normalized test. EOC is 1.0 by construction for ordered tasks and fails on the two-key control. EOCg is essentially an estimate of rho, so its correlations with rho-driven quantities are circular. "Always normalize" dominates the probe rule. Eq. (2) is undeveloped.
A well-controlled implementation study presented as a mechanistic discovery. The conceptual claim reduces to the product rule, the scale-versus-content separation is confounded, and the evidence beyond grid worlds is weak.
Questions for authors
Report joint success minus the product of marginals, with intervals, for both switch variants. Is there any learning effect beyond the product rule?
Do the main results hold with RMS-only normalization, i.e., without mean-centering?
With batch-normalized advantages, where does reward scale enter? Please report critic loss and explained variance under raw versus normalized rewards.
How do the Overcooked numbers compare with published, tuned PPO baselines?