acceptodds
Under review as a conference paper at ICLR 2027

Measuring Counterfactual Credit in a Replayable Wildfire-Suppression Benchmark

Abstract

Predicting a team's return and assigning credit to individual actions are distinct tasks. FLARE, a stochastic wildfire-suppression benchmark, uses coupled-draw replay to measure immediate placement effects for training and long-horizon action contrasts for grading critics. In an eight-seed MAPPO comparison, simulator credit reduces interquartile mean burned area from 26.6% with shared reward to 20.9%. A fixed non-replay local reward scores 24.7%; simulator credit wins every paired seed against both. Informed learned credit scores 27.0% and meets a prospectively fixed -point equivalence criterion against shared reward. Four informed critics paired with responsive main-comparison actors ( saturated audited decisions) attain factual-return , yet raw credit RMSE is times zero's. Matching critic credit to the finite replay reference's RMS still leaves squared error at times zero's; ten earlier-rule critics show . Held-out MSE fitting instead attenuates current predictions by over 97%, approaching zero-credit error. Current critics also give heterogeneous agents nearly identical action rankings (median Spearman ), with largest-magnitude credits on below-1%-probability actions in rows. These measured structures complement the finite-reference calibration diagnostic. Coupled replay separates feedback performance, numerical scale and action-level structure while quantifying replay precision.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, credit assignment, counterfactual credit assignment, difference rewards, cooperative multi-agent reinforcement learning, differentiable simulation, reinforcement learning benchmark, wildfire suppression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zurfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FLARE is a replayable wildfire-suppression environment for studying multi-agent credit assignment. The paper uses one-step placement-deletion credit for training and a separate long-horizon replay reference to audit COMA-style critics. Simulator credit improves suppression, while the audited critics show large raw credit errors despite accurate return prediction.

Critical strengths

The matched MAPPO comparison is convincing: simulator credit beats both shared reward and the non-replay local reward in all eight paired training seeds. This is stronger evidence than a comparison against shared reward alone.

The critic audit makes a useful distinction between predicting returns and assigning action credit. Reporting scale calibration and replay uncertainty helps identify what the observed errors actually establish.

Critical weaknesses

The long-horizon audit uses eight contexts per critic and 16 replay draws per action. Reference uncertainty is comparable to the credit magnitude, with poor agreement between independent replay halves. The raw scale error is convincing, but action-ranking and gradient-direction conclusions remain inconclusive.

Four of eight current-rule learned-credit actors saturate, while the no-clamp controls use earlier simulator rules. This leaves the lack of learning improvement partly entangled with the actor implementation.

Questions for authors

Do higher-budget replays, particularly earlier in episodes, produce stable action rankings and preserve the findings after scale calibration?

Does learned credit still fail to improve performance when agents are retrained under the current rules without the logit clamp?

2 comments
AuthorsOpenReview ↗

**Response to Reviewer zurf (1/2)**

Thank you for the clear summary and for two precise concerns. We addressed both with new experiments:

  • retraining learned credit under the current rules without the logit clamp (8 seeds), also with the critic trained on replayed outcomes of untaken actions (W2, Q2; Section 6.2);
  • replaying every action 64 times in the original contexts and in 8 new, earlier ones for all four audited critics, and 256 times for one critic (W1, Q1; Section 6.3).

In short, removing the clamp lets most actors keep learning, because an actor saturates only when the clamp cuts off its gradient (Section 4). Learned credit still does no better than the shared reward, most likely because the random fire moves the team's return far more than one agent's choice does, so a critic that sees the return only after the action actually taken learns little about how an agent's actions compare (Sections 1 and 5). For the same reason, the replay credit of each action is small next to the noise of the replay itself, which is why more replay draws strengthen the raw scale finding (post 2) and why ranking an agent's actions needs a larger replay budget (Section 6.3). We revised the main text accordingly. Section numbers refer to the revised PDF, and Appendix J collects the new results in three tables.

W2 and Q2. Saturated actors and the no-clamp control.

"Four of eight current-rule learned-credit actors saturate, while the no-clamp controls use earlier simulator rules."

"Does learned credit still fail to improve performance when agents are retrained under the current rules without the logit clamp?"

Yes, it still fails to improve on the shared reward. We retrained learned credit under the current rules without the clamp, with everything else unchanged, and compared it with the shared reward and simulator credit retrained in the same round with the same seeds. The training check, fixed before training, flags an actor whose actions barely depend on what it observes or that still acts almost at random.

Current rules, 8 seedsFail training checkBurn score (%, lower is better)Worse than shared reward
Learned credit, clamp removed1 of 827.38 of 8 pairs
Learned credit, clamp removed, + replay labels3 of 826.98 of 8 pairs
Shared reward0 of 826.6
Simulator credit0 of 819.3

(Interquartile mean over 8 seeds; all rows were retrained in the same round of experiments, so they compare directly.) Without the clamp, seven of the eight actors pass the check (in the submitted runs with the clamp, four of eight saturated). Yet learned credit burns more than the shared reward in all eight seed pairs (by 0.65 points, 95% interval [0.34, 0.91]), and simulator credit beats it by 7.9 points [6.6, 8.9]. Training the critic on replay labels for untaken actions, as Reviewer auGw suggested, does not change this, and three of those eight runs fail the training check. So the lack of improvement does not come from the clamp, and it holds with actors that keep learning. Section 6.2 reports these runs and states the result as learned credit doing no better than the shared reward.

AuthorsOpenReview ↗

**Response to Reviewer zurf (2/2)**

W1. Eight contexts and 16 draws.

"Reference uncertainty is comparable to the credit magnitude, with poor agreement between independent replay halves. The raw scale error is convincing, but action-ranking and gradient-direction conclusions remain inconclusive."

We agree with this reading, and the new replays sharpen both parts of it. The raw scale error becomes even more convincing as replays are added, and the replay uncertainty you point to is informative in itself. In each replay, all 20 actions of an agent share the same random draws, so the differences between them come from the agent's choice. Even so, the effect of that choice varies so much from one random future of the episode to the next that its average, the credit, emerges only over many replays (Q1). This supports the reason given in post 1 for the failure of learned credit: if hundreds of replays are needed to rank one agent's actions, a critic that sees one return, for the one action taken, has far less to go on (Section 5).

The audit therefore rests on two findings that replay noise cannot create. The first is the raw scale error. Noise can only make the replay credit look larger, so comparing the critic's credit with it can only understate how inflated the critic's credit is, and the gap widens as replays are added (Q1). The second is that each critic ranks the 20 actions almost identically for every agent (median Spearman 0.98 to 0.99). This is computed from the critic alone, and untrained critics of the same design already show it. The action ranking itself, and the gradient direction and the comparisons after scale calibration that build on it, remain open until a larger replay budget is available (Section 6.3, Appendix H).

Q1. More draws, earlier contexts and scale calibration.

"Do higher-budget replays, particularly earlier in episodes, produce stable action rankings and preserve the findings after scale calibration?"

For all four critics, we replayed every action 64 times at 8 new, earlier contexts (steps 10 to 48) and at the original 8, so each critic now has 16 contexts (Appendix J). For seed 8, the critic closest to our precision threshold at 64 replays, we replayed its early contexts 256 times.

  • Raw scale finding: it holds and grows. We measure the size as the critic's credit error divided by the error of giving every action zero credit, which says roughly how many times larger the critic's credit is than the replay credit. At the original contexts, this ratio rises from 9 to 16 with 16 replays per action to 20 to 31 with 64. At the early contexts of seed 8, it rises from 20 with 16 replays to 34 with 256.
  • Action rankings: precision grows with the budget. We measure precision by the signal-to-noise ratio: the root mean square (RMS) of the replay credit divided by its Monte Carlo error. The measured RMS contains the noise as well as the signal, RMS2≈signal2+noise2\mathrm{RMS}^2 \approx \mathrm{signal}^2 + \mathrm{noise}^2, so a ratio of 1 means the replay credit could be pure noise, and signal equals noise only when the ratio reaches 2≈1.41\sqrt{2} \approx 1.41. Before running, we required the lower end of the 95% interval to reach 1.41. At 256 replays, seed 8 reaches 2.0 (interval 1.1 to 3.0), and two of its 8 contexts are already clean on their own (ratio of at least 2, with independent replay halves agreeing). Stable rankings across all contexts need a larger budget still.
  • Earlier contexts: slightly more precise. At 64 replays, the ratio is 1.1 to 1.4 at the early contexts and 0.9 to 1.2 at the late ones.
  • Findings after scale calibration: they follow the ranking. Once the scale is removed, these comparisons depend only on the order of the actions' credit (Section 5), so they become gradable together with the ranking (Section 6.3, Appendix H).

Summary. Your two points lead to three findings. The raw scale error holds at every budget and context, and it grows as replay noise falls. The near-identical rankings across agents need no replay reference at all. Ranking one agent's actions needs more than the 256 replays per action we ran, which measures how small one agent's effect is next to the randomness of the fire: the condition under which a critic can predict returns well and still give the wrong credit (Section 1).

Thank you again for the precise comments; they led directly to the no-clamp retraining and to the larger replay budgets.

Reviewer_auGwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FLARE is a replay-enabled multi-agent wildfire suppression benchmark with heterogeneous agents. Using deterministic state resets and coupled random draws, it provides one-step counterfactual credit for training and long-horizon replay references for auditing critics. MAPPO experiments show that one-step simulator credit outperforms shared rewards, non-replay local rewards, and COMA-style learned credit. Notably, replay audits and theoretical analysis reveal that centralized critics can fit factual team returns well while producing severely miscalibrated counterfactual credit—consistently performing worse than a simple zero predictor.

Critical strengths
  1. Strong benchmark for credit assignment: Coupled sampling reduces one-step credit variance by 934x compared to independent replay and removes spurious negative credit. Preregistered evaluation and paired-seed tests make the results highly reliable.

  2. Clear gap between return fitting and credit quality: In the current-rule audit, critics achieve high factual return fit ($ R^2 > 0.91 $) but output nearly identical action rankings across different agents (median Spearman correlation of $ 0.980\text{--}0.988 $).

  3. Solid theoretical grounding: The non-identifiability proposition clearly explains why fitting factual returns cannot constrain unsampled single-agent actions in large joint-action spaces.

  4. Informative local-reward baseline: Including a non-replay local reward rules out the possibility that simulator credit succeeds merely by providing dense, per-agent feedback.

Critical weaknesses
  1. Over-simplified wetline mechanics: In the example episode (Figure 9), ground units mostly place single-cell wetlines because they require neither an anchor nor water. This reduces tactical realism and makes myopic one-step optimization artificially effective.

  2. Noisy long-horizon audit references: With $ M=16 $ coupled replays, the Monte Carlo error is as large as the reference signal itself. While this is enough to show that critic predictions are inflated by an order of magnitude, evaluating fine-grained action rankings requires cleaner references.

  3. Myopic training credit: The one-step deletion credit $ d_{it}^{\text{place}} $ only captures immediate fire-blocking effects, ignoring preparatory movement or multi-step anchor building.

  4. No tested fixes for the critic: The paper diagnoses why centralized critics fail but does not test whether adding a few replay queries or simple regularizers can fix the issue.

Questions for authors
  1. If ground wetlines had water and refill limits, would one-step simulator credit still beat local and shared rewards,especially when anchored firelines require multi-step planning?

  2. Have you tried using a small budget of coupled replays as an auxiliary loss on $ q_\phi $ to anchor predictions for unsampled actions?

  3. Seven of the eight query steps (28 of 32 audit contexts) happen late in the episode ($ t \ge 55 $). Do earlier steps (e.g., $ t < 30 $) have a better signal-to-noise ratio or better critic alignment?

  4. Do the nearly identical cross-agent action rankings happen because the shared critic backbone ignores focal-identity and agent-type features? Would separate role-specific heads help?

3 comments
AuthorsOpenReview ↗

**Response to Reviewer auGw (1/3)**

Thank you for the careful and constructive review. Each of your questions led us to a new experiment:

  • retraining under more realistic rules, in which ground crews carry and refill water, and firelines may also start from burned ground (W1, Q1; Section 6.2);
  • replaying every action 64 times at earlier query steps for all four audited critics, and 256 times for one critic (W2, Q3; Section 6.3);
  • the two critic repairs you proposed, replay labels for untaken actions and one output head per agent type, plus removing the clamp on the policy's logits (W4, Q2, Q4; Section 6.2).

In short, simulator credit wins every seed pair under each rule change, and its lead is largest when both changes are combined. With more replays, the critics' credit looks even more inflated, and grading action rankings needs a larger replay budget. Neither repair makes learned credit better than the shared reward. All new training runs use 8 seeds unless stated otherwise. Section numbers refer to the revised PDF, and Appendix J collects the new results in three tables.

W1. Over-simplified wetline mechanics.

"ground units mostly place single-cell wetlines because they require neither an anchor nor water. This reduces tactical realism and makes myopic one-step optimization artificially effective."

Ground crews (hand crews and dozers) have three suppression tools (Section 3 and Appendix A), which act on the fire in different ways:

  • Wetline puts out the burning in one cell and makes that cell harder to ignite for one step; the wetting then clears. It needs no anchor and, under the current rules, no water.
  • Scratchline halves the remaining fuel of one cell. The change lasts for the rest of the episode, but it only weakens the fire there. It needs no anchor.
  • Fireline clears all fuel from one cell (up to three for a dozer), a permanent barrier that the fire cannot burn, but it must start from the map edge or an existing fireline.

A wetline acts at once, so one-step credit sees its effect right away, while a fireline's value builds up over many later steps, which one-step credit sees only in part. Free, unanchored wetline can therefore favor one-step credit. To test how much, we retrained under two rule changes (Section 6.2). With limited water, each ground crew carries 8 wetline uses, and a crew with an empty tank cannot place wetline until a 5-step refill ends, as for aircraft. With firelines from burned ground, a fireline may also start next to burned ground, so crews can build lines outward from the burned area over several steps. Burn score (each cell's burning time, capped and averaged over the map, in %; interquartile mean over 8 seeds; lower is better):

RulesShared rewardLocal rewardSimulator creditSimulator credit wins
Current rules26.625.019.38/8 pairs vs both
Limited water26.825.121.78/8 pairs vs both
Firelines from burned ground23.1not trained14.98/8 pairs vs shared
Both changes21.217.511.38/8 pairs vs both

The first row repeats the main comparison, retrained in the same round of experiments as the rule changes so that all rows compare directly. Simulator credit keeps its lead in every seed pair under each change. Limited water shrinks the lead over the shared reward from 7.3 to about 5 points (95% interval [4.3, 5.6]), which agrees with your point that free wetline helped, and with both changes the lead grows to 9.9 points [7.2, 12.3], the largest we measured (Section 6.2). The local reward also beats the shared reward in every seed pair under each rule set where we trained it, so per-agent feedback helps, and simulator credit helps further.

AuthorsOpenReview ↗

**Response to Reviewer auGw (2/3)**

Q1. Water limits and anchored firelines.

"If ground wetlines had water and refill limits, would one-step simulator credit still beat local and shared rewards, especially when anchored firelines require multi-step planning?"

Yes. Simulator credit beats the shared reward in all 8 seed pairs under every rule set we tested, and the local reward under every rule set where we trained it (table under W1). On anchored firelines: under the current rules a fireline already needs an anchor (the map edge or an existing fireline), and we now state this rule precisely in Section 3 and Appendix A.1. The second change also accepts burned ground as an anchor, which gives crews more places to start a line and makes building a line over several steps next to the fire practical. Simulator credit's lead over the shared reward is 8.2 points [4.9, 12.2] with this change alone and 9.9 points [7.2, 12.3] with both changes.

W2. Noisy long-horizon references.

"With M=16 coupled replays, the Monte Carlo error is as large as the reference signal itself. [...] evaluating fine-grained action rankings requires cleaner references."

We agree. The new replays make the size finding stronger, and, as you anticipated, ranking actions needs cleaner references than even 256 replays per action provide. In your notation, let A^\hat A be an action's replay credit averaged over MM replays and MCSE its Monte Carlo standard error. Then RMS(A^)2≈S2+MCSE2\mathrm{RMS}(\hat A)^2 \approx S^2 + \mathrm{MCSE}^2, where S2S^2 is the signal and the noise MCSE2\mathrm{MCSE}^2 falls as 1/M1/M. Our precision threshold uses SNR=RMS(A^)/MCSE\mathrm{SNR} = \mathrm{RMS}(\hat A)/\mathrm{MCSE}: the lower end of its 95% interval had to reach 2≈1.41\sqrt{2} \approx 1.41, the point where signal equals noise.

For all four audited critics, we replayed every action M=64M = 64 times at 8 new, earlier query steps (steps 10 to 48) and at the original 8. For one critic (seed 8, chosen after the 64-replay results because it was closest to the threshold), we then used M=256M = 256 at the earlier query steps. Its SNR rises to 2.0 (95% interval [1.1, 3.0]), still below 1.41 at the lower end. We therefore grade the size of the credit, which replay noise can only understate, while action rankings, and the check of gradient direction that depends on them, remain open until a larger replay budget is available (Sections 5 and 6.3, Appendices H and J).

The size finding gets stronger as noise falls. We measure size by RMSE(Ac,A^)/RMS(A^)\mathrm{RMSE}(A^c, \hat A)/\mathrm{RMS}(\hat A), where AcA^c is the critic's credit; the denominator is the error of the simple zero predictor. When AcA^c is much larger than A^\hat A, this ratio says how many times larger it is. The denominator still contains the noise, so more replays shrink it and raise the ratio:

Replays per actionOriginal 8 query steps8 earlier query steps
169 to 1611 to 23
6420 to 3120 to 42
256 (seed 8 only)not run34 (20 with 16 replays)

(Ranges cover the four critics.)

W3. Myopic training credit.

"The one-step deletion credit only captures immediate fire-blocking effects, ignoring preparatory movement or multi-step anchor building."

One-step credit does reward preparatory moves and line building, through the return; what it misses is the lasting value of a line once it is placed.

Simulator credit measures the burn prevented in the next step: for agent ii at step tt, dit=r(xt,Pt,ξt)−r(xt,Pt−i,ξt)d_{it} = r(x_t, P_t, \xi_t) - r(x_t, P_t^{-i}, \xi_t), the team reward with all accepted placements PtP_t minus the team reward without agent ii's placement, both under the same random draw ξt\xi_t (Equation 3, Section 4). Because ditd_{it} is agent ii's reward in MAPPO, its return ∑k≥0γkdi,t+k\sum_{k \ge 0} \gamma^k d_{i,t+k}, and the GAE advantage built from it, credit an earlier action through the suppression it makes possible later. A move toward the fire is credited through the drops or lines that follow, and so is a fireline segment that blocks nothing yet but lets the same crew anchor a later segment that does. What one-step credit does not capture is a segment's own blocking value in later steps: a fireline is credited only for the step in which it is placed.

Two results suggest that this limit did not remove the advantage of simulator credit when lines matter. When firelines may also start from burned ground, so that lines can be built over several steps, its lead over the shared reward stays similar (8.2 points against 7.3 under the current rules). With both rule changes, when wetline is also limited, the lead is the largest we measured (9.9 points; post 1). Credit over several replayed steps, which FLARE's replay already supports, would also capture a line's lasting effect (Section 7).

AuthorsOpenReview ↗

**Response to Reviewer auGw (3/3)**

W4 and Q2. Replay labels and a simple regularizer for the critic.

"does not test whether adding a few replay queries or simple regularizers can fix the issue."

"Have you tried using a small budget of coupled replays as an auxiliary loss on q to anchor predictions for unsampled actions?"

Yes: following your suggestion, we trained the critic with replay labels, and they do not make learned credit better than the shared reward (Section 6.2 and Appendix J). In each update, we replay 4 of the 19 untaken actions at each of 128 random agent steps, one step ahead (512 replays), under the same random draw as the step that happened and with the teammates' actions fixed. For an untaken action bb at a step where the agent took aa, the label is yb=Gt+(rb−ra)+γ(vb−va)y_b = G_t + (r_b - r_a) + \gamma (v_b - v_a), where GtG_t is the critic's usual target for aa, rb−rar_b - r_a is the difference in that step's reward, and vb−vav_b - v_a is the difference in the critic's values at the two next states. The squared error to yby_b is added to the critic's loss. Learned credit is trained under the current rules without the clamp on the policy's logits.

Training signalSeedsBurn score (%)Worse than shared reward
Shared reward826.6
Simulator credit819.3
Learned credit827.38 of 8 pairs
+ replay labels826.98 of 8 pairs
Learned credit, three-seed trial327.23 of 3 pairs
+ simple regularizer, no replay327.13 of 3 pairs
+ one output head per agent type327.23 of 3 pairs
+ both repairs326.93 of 3 pairs

(Interquartile mean for 8 seeds, mean for 3 seeds; the first two rows repeat post 1.) Replay labels lower the burn score by only 0.4 points (95% interval 0.0 to 0.7), and learned credit stays 7.6 points behind simulator credit. A simple regularizer of the kind you mention does no better (27.1% in the three-seed trial): it uses GtG_t as the label for the same untaken actions, with no replay, which pulls their values toward the target of the action taken. The likely reason is that replay labels carry little more than this regularizer: in the three-seed trial, the untaken action changed that step's reward in at most 3% of these replays, so most labels differ from GtG_t only by γ(vb−va)\gamma (v_b - v_a), which comes from the critic's own values. Labels from replays over several steps may carry more (Section 7).

Q3. Earlier query steps.

"Seven of the eight query steps (28 of 32 audit contexts) happen late in the episode. Do earlier steps have a better signal-to-noise ratio or better critic alignment?"

Earlier steps are not clean enough either, so the critics' alignment still cannot be graded there. With 64 replays per action at the 8 earlier query steps (post 2; Section 6.3), the signal-to-noise ratio is 1.1 to 1.4 early and 0.9 to 1.2 late, below our threshold in both. Late steps differ mainly in how often no action changes the outcome in any replay (5 to 44 of each critic's 64 agent decisions, against 0 to 6 early), yet the critics' credit is as inflated early as late (table in post 2).

Q4. Identical rankings and role-specific heads.

"Do the nearly identical cross-agent action rankings happen because the shared critic backbone ignores focal-identity and agent-type features? Would separate role-specific heads help?"

The rankings barely depend on these features, and this is already true before training. The critic receives each agent's identity and type, yet untrained critics of the same design rank actions almost identically across agents (median Spearman 0.98 to 0.99; Section 6.3). Training does not change this, as expected from Section 5: fitting the returns of the actions taken leaves the values of untaken actions undetermined. In the three-seed trial, each agent type got its own copy of the critic's output layer. Heads alone left the rankings nearly identical in two of the three runs, while heads combined with replay labels, which give the critic targets for untaken actions, separated them: the Spearman correlation between agents' rankings fell from about 0.98 to between 0.14 and 0.30. Yet learned credit still did worse than the shared reward in every seed pair (table above). Although several actors in this trial ended up repeating a single fixed action, the trial gives no sign that separate rankings alone repair learned credit.

Thank you again for these constructive suggestions. We hope the new experiments address your concerns.

Reviewer_BwPcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors explore whether replacing shared team rewards with per-agent credit assignment improves learning in cooperative AI tasks. They also ask whether centralized critics are "sufficient" for capturing per-agent credit assignment. They explore this in a wildfire simulation benchmark, FLARE. To answer whether replacing the shared team reward improves learning they compute "simulator credit", which as I understand it is training counterfactually on how much a single agent's choice of area to suppress the wildfire affected to the simulation in terms of wildfire spread. They find about a 6% imrovement by training on simulator credit versus just a shared reward, and a 4% improvement compared with a fixed local proximity reward. They also test centralized critics and conclude they perform about the same as the shared reward. For the second question on whether centralized critics that predict actions better also get per-action credit correct, they find this is not the case. The critics fit the return really accurately statistically, but have substantially higher error and different agent types get the same near-identical action rankings.

Critical strengths
  • The experiments of the paper are clean and clear and mostly cleanly show the authors' hypothesis.
  • The flare experiment and setting is very nice
  • The math and statistical analysis is very nicely done
Critical weaknesses
  • Major citations missing? Learned from difference of re-simulated rewards is well established in a number of papers that seem to do this already that are not cited. Tumer and Agogino 2007? CLEAN rewards (HolmesParker et al 2007), Colby et al 2015, D++ 2016 (Rahmattalabi et al 2016) all seem to do similar approaches and are not cited. Even TRPO Schulman et al 2015 does something conceptually similar with its "vine" procedure. So there is a major question as to whether this approach is an innovation and if not, then what is the main contribution?
  • There are some questionable things in the experiments. For example, how much noise is there for each of the computed statistics? What is the maximum best any centralized-critic or agent-specific credit assignment can do and seeds 2 and 3 get identical rewards on all 30 fires.
  • Writing is incredibly hard to parse. To the point that most of the content I gave, as I understand it is in the summary. At least to me as someone relatively familiar with this subarea that was completely unparseable and I gave up and asked a LLM to restate what it was saying for me. It understood it perfectly on the first go. I suspect this may be a consequence of using LLM-generated text that "overcompresses", drops intermediate logical steps/arguments and over and incorrectly uses technical terms in place of standard terminology. This continues throughout the paper. Effectively, to be readable to a human, I think the paper probably has to be rewritten.

The remaining questions are mostly about what the main contribution of the paper is vis-a-vis the prior literature.

I think realistically the paper needs to be rewritten before it is ready for submission. Once that is done, then I think it is worthwhile considering its contribution... I like the fundamental idea but I am not sure that the insight that individual credit assignment helps in a coupled-performance task is that surprising. I am not sure of this, though, and willing to be convinced of this, though. Without the novelty of the method (which would need to be established against prior literature) I am not sure the contribution is so clear.

Questions for authors
  • What is the main innovation of the paper compared to the prior literature?
  • If prior work has done the counterfactual analysis, then is it mainly about how much credit assignment effects agents? I don't think this result is that surprising, though, that using a centralized critic leads to shared credit assignments. Is the point to highlight this and its performance impacts or to emphasize something more subtle?
  • What is the noise/uncertainty and theoretical maximum you can do on the experiments for different critic types.
3 comments
AuthorsOpenReview ↗

**Response to Reviewer BwPc (1/3)**

Thank you for your review and for the pointers to prior work. We appreciate your comments on readability and novelty, and in these three posts we explain our contributions and the changes we made to the paper. This post covers readability (W3) and whether the finding is surprising (W4 and Q2), post 2 covers prior work and the main innovation (W1 and Q1), and post 3 covers noise, the best possible score and seeds 2 and 3 (W2 and Q3). Section numbers refer to the revised PDF.

W3. Readability.

"Writing is incredibly hard to parse. [...] drops intermediate logical steps/arguments and over and incorrectly uses technical terms in place of standard terminology."

We agree that parts of the submitted text were hard to parse. To fit the page limit, we compressed the argument and left out intermediate steps. We have now gone carefully through the whole manuscript, sentence by sentence, and improved the wording wherever it was hard to parse, especially in the introduction and the related work (Sections 1 and 2). We also replaced several technical terms with the standard ones, and we used the extra page that the revision allows to spell out the intermediate steps and to explain the method more fully. We hope the paper now reads much more easily.

These changes affect only how the paper explains its results: no numerical result and none of the main equations from the submission have changed. Our effort went into making each step easier to follow, from rerunning a fire in the simulator to training the agents and checking the critics. Sections 6.2 and 6.3 also report new experiments that Reviewers auGw and zurf requested.

W4 and Q2. Is the finding surprising?

"I am not sure that the insight that individual credit assignment helps in a coupled-performance task is that surprising."

"Is the point to highlight this and its performance impacts or to emphasize something more subtle?"

Individual credit helping here is indeed expected, and our results confirm it. With MAPPO held fixed, even a simple local reward (paid to each agent for suppression next to the fire) beats the shared reward in all 8 seed pairs. The burn score (each cell's burning time, capped and averaged over the map; lower is better) falls from 26.6% with the shared reward to 24.7% with the local reward and 20.9% with simulator credit, our difference reward computed by replaying each step under the same random draw, which wins all 8 seed pairs against both (Section 6.2). The revision presents this as a confirmation of earlier work (Castellini et al., 2025). What surprised us was learned credit, the credit that a trained COMA-style critic gives each action. We expected critics that predict the team's return well to give roughly correct credit, but our critics predict it with R2R^2 of about 0.9, and replaying each action to the end of the episode shows that the credit they give individual actions is still 9 to 16 times too large (Section 6.3). Although this credit, like the local reward, is given to each agent separately, agents trained with it do no better than with the shared reward.

The subtler point is that a good fit to the team's return does not validate a critic's credit. The ratio of 9 to 16 is the critics' credit error divided by the error of giving every action zero credit, and the error of zero credit equals the size of the replay credit, so the ratio says roughly how many times too large the critics' credit is. Replay noise makes the replay credit look larger, which lowers the ratio, so more replays raise it. The finding also goes beyond a shared critic giving shared credit: the critics receive each agent's identity and type, yet they rank the 20 actions almost identically for hand crews, dozers and aircraft, as untrained critics of the same design already do (Section 6.3). This second finding uses the critics alone, so it does not depend on replay noise. Neither problem shows up in the usual check, how well a critic predicts the team's return.

AuthorsOpenReview ↗

**Response to Reviewer BwPc (2/3)**

W1 and Q1. Prior work and the main innovation.

"So there is a major question as to whether this approach is an innovation and if not, then what is the main contribution?"

"What is the main innovation of the paper compared to the prior literature?"

Thank you for these five references. We read all of them, and the revision cites each one in Sections 1 and 2. We agree that the method itself is established, and the paper presents it that way: difference rewards go back to Wolpert and Tumer (1999), and the submission's introduction called difference rewards and common random numbers established tools. For each paper, we say below what it did, what changes in FLARE, and how our findings compare.

Tumer and Agogino (2007) gave each agent a difference reward by rerunning their air traffic simulator for the whole episode with that agent's action removed; their agents act once per episode. FLARE needs the same quantity in the middle of an episode, while the fire spreads randomly. A plain rerun would then change two things at once, the agent's action and the random spread, and the random part dominates: with independent random draws, the credit has 934 times the variance and is negative in 47% of samples, although removing a suppression can only add burning (Section 6.4). Replaying the step from its saved state under the same random draw leaves only the agent's effect. Our training result agrees with theirs, now with deep MAPPO and eight agents of three types (Section 6.2).

CLEAN (HolmesParker et al., 2014) works in one-shot games whose team score is a known formula, so an agent can compute exactly what any untaken action would have scored. FLARE has no such formula, so we tested a similar idea through replay: training the critic on replayed outcomes of untaken actions left learned credit no better than the shared reward (26.9%). In a one-shot game, an alternative's full effect appears at once. In our trial, an untaken action changed that step's reward in at most 3% of one-step replays, most likely because its effect arrives later through the spreading fire.

Colby et al. (2015) replaced the formula with a learned estimate of the team's score, and the resulting credit kept 75 to 98% of the benefit of exact credit, judged by final team performance in deterministic one-shot games. We find the opposite for a COMA-style critic in a stochastic task: learned credit does no better than the shared reward, and its credit is 9 to 16 times too large. We think two features explain this. The team's return varies hundreds of times more from one moment to the next than between one agent's actions at a given moment, so a critic can fit returns well (R2R^2 about 0.9) while learning almost nothing about the small differences between actions. And the critic sees outcomes only for the actions actually taken, so nothing pins down the credit of the other 19 actions.

D++ (Rahmattalabi et al., 2016) targets tasks where a reward appears only when several robots act together, so removing one robot often changes nothing and the plain difference reward gives no signal; D++ adds imagined copies of the robot to restore it. Suppression in FLARE is not gated this way: one agent's placement prevents burning on its own, so removing it already gives a signal, consistent with simulator credit beating both the shared and the local reward.

TRPO's vine procedure (Schulman et al., 2015) resets a single agent's simulator to saved states, rolls out several actions with shared random numbers, and trains on the results. We apply the same procedure to one agent inside a team of eight, with its teammates fixed, as a measuring tool. With 16 replays per action, the replay credit is about as large as its own noise, and even 256 replays, which we ran for one critic, were not enough to rank an agent's 20 actions, likely because one agent's choice moves the return far less than the random fire and its seven teammates do. We therefore grade the size of the credit, where replay noise can only make the critics look better than they are.

Castellini et al. (2025), the closest earlier check of a learned critic, compared a trained COMA critic's action values with rollouts and found large errors. Our critics pass that kind of value check (R2R^2 0.91 to 0.95) and still give credit 9 to 16 times too large (Section 6.3). Credit is the small difference between an agent's action values, so a critic that explains 90% of the variation in returns can still get those differences badly wrong, and only a check of the credit itself, with all actions replayed under the same random draws, shows it.

Our main innovation is to check COMA's critic against the extra simulations it was built to replace (Foerster et al., 2018): because FLARE can run them, the size of the credit that a critic gives each action can be measured directly.

AuthorsOpenReview ↗

**Response to Reviewer BwPc (3/3)**

W2 and Q3. Noise, the best possible score, and seeds 2 and 3.

"how much noise is there for each of the computed statistics? What is the maximum best any centralized-critic or agent-specific credit assignment can do and seeds 2 and 3 get identical rewards on all 30 fires."

"What is the noise/uncertainty and theoretical maximum you can do on the experiments for different critic types."

Noise. Each training signal has 8 seeds, and comparisons use paired 95% intervals over seeds, from 10,000 bootstrap resamples or a paired t test where marked (Section 6.1). Critic statistics carry the Monte Carlo error of each replay estimate and intervals that resample whole episodes (Section 5 and Appendix E). To check stability, we retrained all training signals with 8 new seeds each and replayed every action 64 times for the four graded critics, four times the submission's 16 (Section 6.3 and Appendix J).

StatisticSubmissionNew experiments
Burn score, shared reward minus simulator credit (points)5.76 [3.65, 7.73]; 8/8 seed pairs7.3 [5.9, 8.5]; 8/8 seed pairs
Burn score, local reward minus simulator credit (points)3.9 [2.1, 5.7], paired t; 8/8 seed pairs5.7 [4.3, 6.8]; 8/8 seed pairs
Error of learned credit divided by the error of giving every action zero credit9 to 16, every interval above 120 to 31 with 64 replays

(A positive difference means that simulator credit leaves less burning; brackets are 95% intervals.)

Theoretical maximum. For the burn score, the optimum of this stochastic task is unknown, so the paper compares the training signals with each other on the same fires and seeds (Section 6.1). For the critics, the best possible score is that of a perfect critic, one that knows each action's true credit, graded against our noisy replay reference: with 16 replays per action, its cosine similarity with the replay credit would be about 0.5 at best (1.0 is perfect agreement), and near zero for two of the four critics. Such a reference can grade the size of the credit, but not yet the order of actions (Section 6.3 and post 2).

Seeds 2 and 3. These are separate runs, but both policies stopped learning: their action scores (logits) reached the limit we set on them (the logit clamp), and both ended up repeating the same fixed action, so they score the same on all 30 fires. Half of the actors trained with learned credit froze this way. To make sure this did not drive our conclusion, we retrained learned credit without the clamp for all 8 seeds, under the current rules (the paper's default) and in the same round as the other signals (Section 6.2 and Appendix J). Our training check, fixed before training, flags an actor whose actions barely depend on what it observes or that still acts almost at random.

Retrained, 8 seeds eachFail training checkBurn score (%)Worse than shared reward
Learned credit, clamp removed1 of 827.38 of 8 pairs
Same, critic also trained on replays of untaken actions3 of 826.98 of 8 pairs
Shared reward0 of 826.6
Simulator credit0 of 819.3

(Interquartile mean over 8 seeds; in the submitted runs, simulator credit scored 20.9%.) Without the clamp, most actors keep learning, yet learned credit still does no better than the shared reward. So the clamp explains seeds 2 and 3, but not the failure of learned credit. We think that failure comes from what the critic can learn: it sees the team's return only after the action actually taken, and the random fire moves that return far more than one agent's choice does, so its training data barely constrain how the 20 actions differ, which is exactly what credit measures (Section 5 formalizes this). The near-identical action rankings across agent types, already present before training (post 1), fit this view: learned credit carries little information about each agent's own contribution.

Thank you again for the time you took to read our paper and write this review. Your feedback on the writing, especially that it skipped steps and used technical terms in place of standard ones, was very useful, and we did our best to address it in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

**Rebuttal Summary**

We sincerely thank you for your time and effort in managing the review process for our paper. Below we summarize the strengths the reviewers noted, how we addressed each concern, and our contributions.

Strengths

  • FLARE is a strong benchmark with clean, reliable experiments (mentioned by all reviewers): "Strong benchmark for credit assignment" (auGw); "The flare experiment and setting is very nice" (BwPc); "The matched MAPPO comparison is convincing" (zurf).
  • The critic audit separates predicting returns from assigning credit (auGw, zurf): "Clear gap between return fitting and credit quality" (auGw); "a useful distinction between predicting returns and assigning action credit" (zurf).
  • Sound statistics and theory (BwPc, auGw): "The math and statistical analysis is very nicely done" (BwPc); "Preregistered evaluation and paired-seed tests make the results highly reliable" and "Solid theoretical grounding" (auGw).

Discussion

We strengthened the paper based on the discussion as follows; Appendix J collects all new results in three tables.

Discussion pointBwPcauGwzurfOur response
Readability and standard terms✓1 / Whole paper
Prior work and main contribution✓2 / Sec. 1, 2
Noise in each statistic, best possible score✓3 / Sec. 6, reply to BwPc
Actors that stopped learning (seeds 2 and 3)✓✓4 / Sec. 6.2
Free wetline and one-step credit✓5 / Sec. 3, 6.2
Noisy replay references, earlier steps✓✓6 / Sec. 5, 6.3, App. H
Repairs to the critic✓7 / Sec. 6.2, 6.3
  1. We revised the text sentence by sentence and used the extra page to spell out the intermediate steps of the argument and the method, with standard terms in place of several technical ones. No numerical result and none of the main equations changed.
  2. We now cite all five papers suggested by Reviewer BwPc. Difference rewards and common random numbers are established tools; our main innovation is to check a trained COMA-style critic against the extra simulations it was built to replace, by measuring the size of the credit it gives each action in a stochastic team task.
  3. We report the uncertainty of each key statistic. Retrained with 8 new seeds, simulator credit again beats the shared reward (by 7.3 points [5.9, 8.5]) and the local reward (by 5.7 points [4.3, 6.8]) in all 8 seed pairs. For the critics, we report the best score a perfect critic could reach against the noisy replay reference.
  4. Learned credit does no better than the shared reward even when the actors keep learning. Retrained without the logit clamp, 7 of 8 actors pass our training check, yet learned credit burns more than the shared reward in all 8 seed pairs (27.3% against 26.6%).
  5. Simulator credit keeps its lead under more realistic rules. With limited water, with firelines that may start from burned ground, and with both changes, it beats the shared reward in all 8 seed pairs; with both changes, it scores 11.3% against 21.2%.
  6. The critics' credit looks even more inflated as replay noise falls. With 64 replays per action at earlier moments, its error is 20 to 42 times the error of giving every action zero credit, against 9 to 16 times with 16 replays. Ranking an agent's actions needs a larger replay budget than we ran, which shows how small one agent's effect is next to the randomness of the fire.
  7. Neither critic repair proposed by Reviewer auGw makes learned credit better than the shared reward: training on replayed outcomes of untaken actions (26.9%) or one output head per agent type (three-seed trial). The near-identical rankings across agent types are already present before training.

Contributions

We believe FLARE makes learned credit testable against the simulations it replaces. Any moment of a stochastic wildfire can be replayed with different actions under the same random events, which yields exact one-step difference rewards for training and a replay reference for grading the credit of learned critics.

With MAPPO held fixed, simulator credit lowers the burn score from 26.6% to 20.9% and beats both the shared and the local reward in every seed pair, also under more realistic rules. The grading shows what the usual return-fit check misses: critics that predict the team's return with R2R^2 of about 0.9 still give credit 9 to 16 times too large.

Beyond this paper, FLARE gives the community a benchmark on which new credit assignment methods can be checked directly against replay, in addition to final team performance.

Best regards,

Submission 51191 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.