acceptodds
Under review as a conference paper at ICLR 2027

Conjunctive Amplification of Reward-Scale Miscalibration in Cooperative Multi-Event Reinforcement Learning

Abstract

Dense reward shaping is widely used in cooperative MARL, but in some multi-event tasks it can cause catastrophic performance loss. Reward-scale sensitivity is well known in deep RL; what remains unclear is why an ordinary scale mismatch becomes catastrophic only for particular task structures. We show that the missing link is conjunctive amplification. Miscalibrated rewards moderately reduce the acquisition rates of individual necessary events. When success requires all of those events, the local losses compound: all-events success closely tracks the product of per-event acquisition rates (). In a controlled Partial Relay environment, increasing dependence on the multi-event chain progressively narrows the empirically safe range across the tested cooperative reward weights, from all six values at to one at (225 runs; interaction ). Running reward calibration substantially mitigates this structure-amplified performance loss, with gains increasing from to as rises, and it generalizes across architectures and benchmarks, including a fourfold increase in Overcooked deliveries (). We further show why static tuning is unreliable: the effective reward scale changes during acquisition, so a good fixed scale can be identified only in hindsight. Finally, we introduce Event Order Consistency (EOC) as a low-cost behavioral descriptor of relay-family structure that can support calibration assessment before full training. These results support calibrating cooperative shaping when multi-event task structure can amplify small learning losses.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, reward shaping, reward calibration, multi-event tasks, conjunctive amplification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XEC2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why dense cooperative shaping can collapse performance in MARL. The claim is that reward-scale miscalibration moderately lowers the acquisition rate of each necessary event, and that tasks requiring all events to succeed multiply these losses ("conjunctive amplification"). Evidence comes from a Partial Relay grid (rho x lambda, 225 runs), an additive-versus-conjunctive switch task (joint success tracks the product of marginals, r = 0.998), and relay tasks where running reward normalization recovers most of the loss. Overcooked, SMACv2, PCGrad and static-rescaling controls are also reported. An Event Order Consistency (EOC) probe is proposed to decide when to calibrate.

Critical strengths

The controls are careful: matched-magnitude noise, dose and exposure sweeps, order-free variants, per-stream normalization, and frozen versus running normalizers.

The practical point is useful. A good fixed reward scale exists but can only be identified in hindsight, so running normalization matters in cooperative MARL.

Critical weaknesses

The mechanism is arithmetic, not learning. If success requires all L roughly independent events, success equals the product of the marginals by definition, and r = 0.998 over configurations ranging from 0 to 1 is guaranteed by range. The paper's own switch task shows per-event degradation is the same under additive and conjunctive reward (p > 0.07). So conjunctive structure changes the metric, not what the agents learn. Calling this "the missing link" and "the causal mechanism" is an overclaim.

"Scale, not content" is not cleanly shown. (a) The calibration subtracts a running mean, which adds a per-step bonus or cost and is not policy-invariant in episodic tasks. RMS-only scaling gives 0.686 versus 0.839 with centering. (b) Advantages are batch-normalized in every run, so the policy gradient is already scale-invariant. The effect must enter through the critic, and the paper never shows this. (c) After normalization, cooperative content still hurts in most tasks: normalized minimal beats normalized full on Hidden Gift (0.973 vs 0.845) and Partial Relay at rho = 1 (0.98 vs 0.84). Only Sequential Plate benefits.

The Overcooked baselines (0.42 vs 1.78 deliveries) are far below standard PPO results, so "fourfold" compares undertrained agents. SMACv2 is not a raw-versus-normalized test. EOC is 1.0 by construction for ordered tasks and fails on the two-key control. EOCg is essentially an estimate of rho, so its correlations with rho-driven quantities are circular. "Always normalize" dominates the probe rule. Eq. (2) is undeveloped.

A well-controlled implementation study presented as a mechanistic discovery. The conceptual claim reduces to the product rule, the scale-versus-content separation is confounded, and the evidence beyond grid worlds is weak.

Questions for authors

Report joint success minus the product of marginals, with intervals, for both switch variants. Is there any learning effect beyond the product rule?

Do the main results hold with RMS-only normalization, i.e., without mean-centering?

With batch-normalized advantages, where does reward scale enter? Please report critic loss and explained variance under raw versus normalized rewards.

How do the Overcooked numbers compare with published, tuned PPO baselines?

Reviewer_6iQ2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when reward-scale miscalibration becomes especially harmful in cooperative multi-event reinforcement learning. Its main hypothesis is conjunctive amplification: moderate reductions in learning the individual events needed for success can compound into a much larger reduction in completing the whole task. A controlled Partial Relay environment varies dependence on an event chain and the cooperative reward weight, while order-free and additive/conjunctive switch controls examine alternative explanations. The intervention is running reward calibration rather than a new MARL algorithm. The paper also proposes Event Order Consistency (EOC) as a restricted relay-family diagnostic.

Critical strengths
  1. The study makes a substantial effort to isolate competing explanations. The structure-scale matrix, self-reward scaling, semantically broken reward controls, order-free tasks, and additive/conjunctive switch experiment collectively go beyond simply observing that normalization improves performance. These controls make the task-structure argument substantially more convincing.
  2. The calibration effect survives several meaningful checks. The paper evaluates fixed rescaling, RMS-only scaling, clipping, partial-stream normalization, architecture changes, and longer training budgets, with additional Overcooked results. It also openly reports exploration failures and learner-dependent target scales, which helps establish the limits of the conclusion.
Critical weaknesses
  1. The scope of the multiplicative explanation is unclear. The product of marginal event probabilities fits the switch task, but may not apply when events depend on one another.
  2. The practical benefit of running calibration and EOC needs clarification. Simple clipping and late-frozen normalization also perform well. The EOC probe adds training cost without outperforming normalization by default.
Questions for authors
  1. Under what event-dependence conditions should the product of marginal acquisition rates When should the product of marginal event probabilities predict overall success? How does event dependence affect this explanation?
  2. Why does reward normalization help when PPO already normalizes advantages? Which part of learning mainly benefits, such as critic training?
  3. Given the strong clipping and late-freezing results, when does continued adaptation provide a clear advantage over simpler alternatives?
  4. What benefit does the 500-episode EOC probe offer over always normalizing? How does it handle cases with no successful probe episodes?
Reviewer_M5VTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines why poorly scaled cooperative rewards can cause large performance drops in tasks that require several events to succeed. Controlled relay and switch experiments suggest that modest losses in individual event acquisition are amplified at the task level by conjunctive structure. Running reward normalization mitigates these failures, with supporting results in Overcooked and PPO-based learner variants. The paper also studies reward-scale drift during learning and introduces Event Order Consistency as a diagnostic within relay tasks.

Critical strengths
  1. The reward-weight sweeps, scaled self-reward experiments, and controls that alter the meaning of shaping rewards provide useful evidence that reward scale itself can cause the observed failures. The event-level measurements and order-free controls also make the proposed interaction with conjunctive structure more concrete.

  2. Calibration produces substantial gains across the relay tasks and two Overcooked layouts, with additional checks on parameter sharing, critic design, and training budget. These results support a useful practical finding within the tested settings, while the reported failure cases help define its limits.

Critical weaknesses
  1. The central mechanism is not yet cleanly isolated. In the switch experiment, moving from additive to conjunctive rewards also changes reward timing and density (Appendix A.4), so the comparison does not isolate conjunction alone. The gradient analysis also leaves unclear how normalizing the summed reward changes event-specific learning through the critic and advantage estimates. The strong product fit supports the observed amplification pattern, but these gaps limit the explanatory contribution beyond documenting sensitivity to reward scale.

  2. Successful validation remains concentrated in custom tasks and PPO-family learners, with two Overcooked layouts providing the main external evidence. SMACv2 does not directly test the structure–scale interaction, while QMIX fails under every strict-relay treatment and therefore provides no successful replication of the proposed mechanism outside the PPO family. The current results leave its applicability across learner families insufficiently established.

Questions for authors
  1. What evidence in the existing controls separates the effect of conjunction from the changes in task-reward timing and density in the switch experiment?

  2. How are the self and cooperative gradients computed when the total-reward normalizer is active, and what do these measurements establish about the role of critic learning or advantage estimation?

  3. What evidence do the SMACv2 and QMIX experiments provide for the proposed structure–scale interaction, given the existing normalization in SMACv2 and the all-failure strict-relay results for QMIX?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.