- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PROWL-2 couples two curricula for model-based cooperative MARL: an IMAC/PLR-style curriculum over start states of imagined rollouts, ranked by positive value loss, and a PROWL-1-style world-model repair curriculum fed by a developer policy that searches the real environment for model failures. The new component is a fidelity gate. The top-K prioritized start states are periodically replayed open-loop through the world model under the recorded actions and compared with the recorded continuation on control-relevant targets (ally death timing, action-mask validity, reward, observation). States above a fixed, pre-set error threshold are quarantined from policy training and their continuations sent for repair; they return after the repaired model passes consecutive audits. Validation: nine SMACv2 scenarios on a MARIE backbone, where PROWL-2 has the highest mean win rate in all nine over 3 seeds, and four MQE tasks on a SeqWM backbone, where it reaches 2.4x and 3.9x the backbone on the two hard tasks. Cumulative ablations show that repair alone helps little, that the ungated curriculum is worse than the backbone everywhere, and that the gate supplies almost all of the gain; gate controls with random or uncertainty-based quarantine do not recover it.
Critical strengths
-
The fidelity gate is a simple, well-motivated fix to a real problem: learning potential measured inside imagination is confounded with world-model error. Its effect is cleanly isolated. All arms branch from a shared per-seed checkpoint, the threshold is fixed a priori and interpretable as a percentile of the backbone's own error distribution, and the gate controls in App. B.3 hold the quarantine count fixed and show that random or uncertainty-based quarantine does not recover the gain. This is why I believe the gate itself, not extra data or extra gradient steps, is responsible for the improvement, and it is the main reason I lean toward acceptance.
-
The negative result that an ungated imagined curriculum falls below the backbone in all nine SMACv2 scenarios and on the hard MQE tasks is useful on its own to anyone porting PLR-style curricula into world models.
Critical weaknesses
-
The necessity and cost of the developer are not established. On SMACv2 the developer adds 1-2x the task-policy interaction (half of it random-policy rollouts used only for the Eq. 4 normalizer) and doubles GPU-hours (Table 7), but is not charged to the 1M budget, so the headline comparisons with MARIE and the model-free baselines are at roughly 2-3x total real interaction. More importantly, quarantined continuations already populate B_WM, so gate + curriculum can run with no developer at all, and this arm is missing. Repair adds at most about 2 points in 8 of 9 scenarios while the gate supplies the rest, so the expensive half of the "dual" curriculum may be unnecessary. This matters because both the paper's framing and its cost hinge on it.
-
The mechanism behind the gain is not pinned down, and the two instantiations differ in ways the text glosses over. At 5v5 the gate rejects only 4.6-7.1% of newly audited states (Table 3) yet beats Dual by 16 points on Terran 5v5. At 10v11 it rejects 50-61% and only 34-67% of quarantines are ever re-admitted, so there the gate mostly vetoes the top of the curriculum rather than returning repaired states, contrary to the Section 4 narrative that the curriculum grows harder as the model improves. The gate controls are run only on Terran 5v5, where rejection is lowest. On MQE the algorithm is materially different: priority is realized TD error on real transitions (i.e. PER, not an imagined curriculum), the gate is a 1-step error against a trailing 95th percentile (the relative-threshold rule that Section 4.3 argues against), and the developer is untrained. The paper's explanation for why Dual hurts, model error inflating imagined learning potential, cannot apply to real-data priorities, yet Dual loses 22 points to Repair on Gate-3. Without a mechanism the result is hard to transfer, and the "same framework on both backbones" claim is overstated.
Questions for authors
- Gate + curriculum with no developer: what are the SMACv2 and MQE numbers? Quarantined continuations already populate B_WM, so this arm needs no extra interaction and is cheaper than every reported arm, so it should be feasible in the rebuttal.
- Per scenario, what share of start states actually sampled for policy updates were (a) quarantined at the time and (b) quarantined and later re-admitted? With PLR rank prioritization at beta = 0.1 the sampling weight is rank^-10, so counts of audited states do not tell me how much of the training distribution the gate touched. Running the App. B.3 gate controls on one 10v11 scenario would answer the same question from the other side.
- Why does Dual collapse on MQE Gate-3 when its priority is real TD error, so the paper's "model error inflates imagined learning potential" explanation cannot apply? What does PER with the same rho do on plain SeqWM?
I would raise my rating to clear acceptance if the first two are answered convincingly, in particular if the no-developer arm shows the developer is needed, or if it does not and the paper is reframed accordingly. Question 3 affects how far I trust the MQE instantiation. Minor clarifications, not affecting the score: how the "95% CI over three seeds" in Table 2 was computed (the widths look like 1.96 x s.e., which is not valid at n = 3), and how SeqWM at its released configuration compares at 1.5M steps.
Follow-up on supporting evidence and clarifications
Dear Reviewer 3pww,
Thank you again for your constructive feedback. As the discussion period draws to a close, we would like to highlight how our response addresses the points you raised:
We will also improve Figure 1’s readability in the manuscript revision.
We hope these analyses and clarifications help resolve your concerns. Could you let us know whether any points remain that you would like us to address before the discussion closes?
Best regards,
The Authors