- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FLARE is a replayable wildfire-suppression environment for studying multi-agent credit assignment. The paper uses one-step placement-deletion credit for training and a separate long-horizon replay reference to audit COMA-style critics. Simulator credit improves suppression, while the audited critics show large raw credit errors despite accurate return prediction.
Critical strengths
The matched MAPPO comparison is convincing: simulator credit beats both shared reward and the non-replay local reward in all eight paired training seeds. This is stronger evidence than a comparison against shared reward alone.
The critic audit makes a useful distinction between predicting returns and assigning action credit. Reporting scale calibration and replay uncertainty helps identify what the observed errors actually establish.
Critical weaknesses
The long-horizon audit uses eight contexts per critic and 16 replay draws per action. Reference uncertainty is comparable to the credit magnitude, with poor agreement between independent replay halves. The raw scale error is convincing, but action-ranking and gradient-direction conclusions remain inconclusive.
Four of eight current-rule learned-credit actors saturate, while the no-clamp controls use earlier simulator rules. This leaves the lack of learning improvement partly entangled with the actor implementation.
Questions for authors
Do higher-budget replays, particularly earlier in episodes, produce stable action rankings and preserve the findings after scale calibration?
Does learned credit still fail to improve performance when agents are retrained under the current rules without the logit clamp?
2 comments
**Response to Reviewer zurf (2/2)**
W1. Eight contexts and 16 draws.
"Reference uncertainty is comparable to the credit magnitude, with poor agreement between independent replay halves. The raw scale error is convincing, but action-ranking and gradient-direction conclusions remain inconclusive."
We agree with this reading, and the new replays sharpen both parts of it. The raw scale error becomes even more convincing as replays are added, and the replay uncertainty you point to is informative in itself. In each replay, all 20 actions of an agent share the same random draws, so the differences between them come from the agent's choice. Even so, the effect of that choice varies so much from one random future of the episode to the next that its average, the credit, emerges only over many replays (Q1). This supports the reason given in post 1 for the failure of learned credit: if hundreds of replays are needed to rank one agent's actions, a critic that sees one return, for the one action taken, has far less to go on (Section 5).
The audit therefore rests on two findings that replay noise cannot create. The first is the raw scale error. Noise can only make the replay credit look larger, so comparing the critic's credit with it can only understate how inflated the critic's credit is, and the gap widens as replays are added (Q1). The second is that each critic ranks the 20 actions almost identically for every agent (median Spearman 0.98 to 0.99). This is computed from the critic alone, and untrained critics of the same design already show it. The action ranking itself, and the gradient direction and the comparisons after scale calibration that build on it, remain open until a larger replay budget is available (Section 6.3, Appendix H).
Q1. More draws, earlier contexts and scale calibration.
"Do higher-budget replays, particularly earlier in episodes, produce stable action rankings and preserve the findings after scale calibration?"
For all four critics, we replayed every action 64 times at 8 new, earlier contexts (steps 10 to 48) and at the original 8, so each critic now has 16 contexts (Appendix J). For seed 8, the critic closest to our precision threshold at 64 replays, we replayed its early contexts 256 times.
- Raw scale finding: it holds and grows. We measure the size as the critic's credit error divided by the error of giving every action zero credit, which says roughly how many times larger the critic's credit is than the replay credit. At the original contexts, this ratio rises from 9 to 16 with 16 replays per action to 20 to 31 with 64. At the early contexts of seed 8, it rises from 20 with 16 replays to 34 with 256.
- Action rankings: precision grows with the budget. We measure precision by the signal-to-noise ratio: the root mean square (RMS) of the replay credit divided by its Monte Carlo error. The measured RMS contains the noise as well as the signal, , so a ratio of 1 means the replay credit could be pure noise, and signal equals noise only when the ratio reaches . Before running, we required the lower end of the 95% interval to reach 1.41. At 256 replays, seed 8 reaches 2.0 (interval 1.1 to 3.0), and two of its 8 contexts are already clean on their own (ratio of at least 2, with independent replay halves agreeing). Stable rankings across all contexts need a larger budget still.
- Earlier contexts: slightly more precise. At 64 replays, the ratio is 1.1 to 1.4 at the early contexts and 0.9 to 1.2 at the late ones.
- Findings after scale calibration: they follow the ranking. Once the scale is removed, these comparisons depend only on the order of the actions' credit (Section 5), so they become gradable together with the ranking (Section 6.3, Appendix H).
Summary. Your two points lead to three findings. The raw scale error holds at every budget and context, and it grows as replay noise falls. The near-identical rankings across agents need no replay reference at all. Ranking one agent's actions needs more than the 256 replays per action we ran, which measures how small one agent's effect is next to the randomness of the fire: the condition under which a critic can predict returns well and still give the wrong credit (Section 1).
Thank you again for the precise comments; they led directly to the no-clamp retraining and to the larger replay budgets.
**Response to Reviewer zurf (1/2)**
Thank you for the clear summary and for two precise concerns. We addressed both with new experiments:
In short, removing the clamp lets most actors keep learning, because an actor saturates only when the clamp cuts off its gradient (Section 4). Learned credit still does no better than the shared reward, most likely because the random fire moves the team's return far more than one agent's choice does, so a critic that sees the return only after the action actually taken learns little about how an agent's actions compare (Sections 1 and 5). For the same reason, the replay credit of each action is small next to the noise of the replay itself, which is why more replay draws strengthen the raw scale finding (post 2) and why ranking an agent's actions needs a larger replay budget (Section 6.3). We revised the main text accordingly. Section numbers refer to the revised PDF, and Appendix J collects the new results in three tables.
W2 and Q2. Saturated actors and the no-clamp control.
Yes, it still fails to improve on the shared reward. We retrained learned credit under the current rules without the clamp, with everything else unchanged, and compared it with the shared reward and simulator credit retrained in the same round with the same seeds. The training check, fixed before training, flags an actor whose actions barely depend on what it observes or that still acts almost at random.
(Interquartile mean over 8 seeds; all rows were retrained in the same round of experiments, so they compare directly.) Without the clamp, seven of the eight actors pass the check (in the submitted runs with the clamp, four of eight saturated). Yet learned credit burns more than the shared reward in all eight seed pairs (by 0.65 points, 95% interval [0.34, 0.91]), and simulator credit beats it by 7.9 points [6.6, 8.9]. Training the critic on replay labels for untaken actions, as Reviewer auGw suggested, does not change this, and three of those eight runs fail the training check. So the lack of improvement does not come from the clamp, and it holds with actors that keep learning. Section 6.2 reports these runs and states the result as learned credit doing no better than the shared reward.