acceptodds
Under review as a conference paper at ICLR 2027

PROWL-2: Fidelity-Gated Exploration for Multi-Agent World Model Dual Curricula

Abstract

World models promise sample-efficient multi-agent reinforcement learning by letting policies learn from imagined interaction. Imagination, however, introduces a fundamental ambiguity: a trajectory with high learning signal may expose a genuine weakness of the task-solving policy, or may simply reflect inaccurate world-model predictions. A curriculum that prioritises such experience without accounting for model fidelity can therefore reinforce model errors rather than improve the agents. We introduce PROWL-2, a fidelity-gated dual-curriculum framework that maintains one curriculum for training the task agents and a second for discovering and repairing world-model failures. PROWL-2 scores imagined experience along two axes—learning potential and model fidelity measured against the real environment—and routes it accordingly: reliable, informative rollouts are promoted to the agent curriculum, while unreliable ones are sent for world-model repair. A dedicated developer policy actively searches for further model failures, and repaired regions are re-evaluated before re-entering agent training, closing the loop between world-model improvement and policy learning. On cooperative tasks in StarCraft Multi-Agent Challenge v2 (SMACv2) and the Multi-Agent Quadruped Environment, spanning discrete combat and continuous robotic collaboration, PROWL-2 consistently outperforms strong model-based and curriculum baselines at matched environment interactions. Our results show that learning inside world models requires deciding not only which imagined experience is useful but which is trustworthy and that making this distinction explicit turns world-model failures into a source of policy improvement rather than a cause of policy error.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, World Models, Curriculum Learning, Model-Based Reinforcement Learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZsFFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PROWL-2 couples two curricula for model-based cooperative MARL: an IMAC/PLR-style curriculum over start states of imagined rollouts, ranked by positive value loss, and a PROWL-1-style world-model repair curriculum fed by a developer policy that searches the real environment for model failures. The new component is a fidelity gate. The top-K prioritized start states are periodically replayed open-loop through the world model under the recorded actions and compared with the recorded continuation on control-relevant targets (ally death timing, action-mask validity, reward, observation). States above a fixed, pre-set error threshold are quarantined from policy training and their continuations sent for repair; they return after the repaired model passes consecutive audits. Validation: nine SMACv2 scenarios on a MARIE backbone, where PROWL-2 has the highest mean win rate in all nine over 3 seeds, and four MQE tasks on a SeqWM backbone, where it reaches 2.4x and 3.9x the backbone on the two hard tasks. Cumulative ablations show that repair alone helps little, that the ungated curriculum is worse than the backbone everywhere, and that the gate supplies almost all of the gain; gate controls with random or uncertainty-based quarantine do not recover it.

Critical strengths
  1. The fidelity gate is a simple, well-motivated fix to a real problem: learning potential measured inside imagination is confounded with world-model error. Its effect is cleanly isolated. All arms branch from a shared per-seed checkpoint, the threshold is fixed a priori and interpretable as a percentile of the backbone's own error distribution, and the gate controls in App. B.3 hold the quarantine count fixed and show that random or uncertainty-based quarantine does not recover the gain. This is why I believe the gate itself, not extra data or extra gradient steps, is responsible for the improvement, and it is the main reason I lean toward acceptance.

  2. The negative result that an ungated imagined curriculum falls below the backbone in all nine SMACv2 scenarios and on the hard MQE tasks is useful on its own to anyone porting PLR-style curricula into world models.

Critical weaknesses
  1. The necessity and cost of the developer are not established. On SMACv2 the developer adds 1-2x the task-policy interaction (half of it random-policy rollouts used only for the Eq. 4 normalizer) and doubles GPU-hours (Table 7), but is not charged to the 1M budget, so the headline comparisons with MARIE and the model-free baselines are at roughly 2-3x total real interaction. More importantly, quarantined continuations already populate B_WM, so gate + curriculum can run with no developer at all, and this arm is missing. Repair adds at most about 2 points in 8 of 9 scenarios while the gate supplies the rest, so the expensive half of the "dual" curriculum may be unnecessary. This matters because both the paper's framing and its cost hinge on it.

  2. The mechanism behind the gain is not pinned down, and the two instantiations differ in ways the text glosses over. At 5v5 the gate rejects only 4.6-7.1% of newly audited states (Table 3) yet beats Dual by 16 points on Terran 5v5. At 10v11 it rejects 50-61% and only 34-67% of quarantines are ever re-admitted, so there the gate mostly vetoes the top of the curriculum rather than returning repaired states, contrary to the Section 4 narrative that the curriculum grows harder as the model improves. The gate controls are run only on Terran 5v5, where rejection is lowest. On MQE the algorithm is materially different: priority is realized TD error on real transitions (i.e. PER, not an imagined curriculum), the gate is a 1-step error against a trailing 95th percentile (the relative-threshold rule that Section 4.3 argues against), and the developer is untrained. The paper's explanation for why Dual hurts, model error inflating imagined learning potential, cannot apply to real-data priorities, yet Dual loses 22 points to Repair on Gate-3. Without a mechanism the result is hard to transfer, and the "same framework on both backbones" claim is overstated.

Questions for authors
  1. Gate + curriculum with no developer: what are the SMACv2 and MQE numbers? Quarantined continuations already populate B_WM, so this arm needs no extra interaction and is cheaper than every reported arm, so it should be feasible in the rebuttal.
  2. Per scenario, what share of start states actually sampled for policy updates were (a) quarantined at the time and (b) quarantined and later re-admitted? With PLR rank prioritization at beta = 0.1 the sampling weight is rank^-10, so counts of audited states do not tell me how much of the training distribution the gate touched. Running the App. B.3 gate controls on one 10v11 scenario would answer the same question from the other side.
  3. Why does Dual collapse on MQE Gate-3 when its priority is real TD error, so the paper's "model error inflates imagined learning potential" explanation cannot apply? What does PER with the same rho do on plain SeqWM?

I would raise my rating to clear acceptance if the first two are answered convincingly, in particular if the no-developer arm shows the developer is needed, or if it does not and the paper is reframed accordingly. Question 3 affects how far I trust the MQE instantiation. Minor clarifications, not affecting the score: how the "95% CI over three seeds" in Table 2 was computed (the widths look like 1.96 x s.e., which is not valid at n = 3), and how SeqWM at its released configuration compares at 1.5M steps.

Reviewer_3pwwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In imperfect world models, useful experiences for policy learning and erroneous experiences generated by the model may produce similar learning signals. To address this problem, the authors propose PROWL-2, which builds on IMAC and PROWL-1 by introducing two curricula and a fidelity gate. The policy curriculum prioritizes training starting points based on their learning potential. The world model curriculum uses a developer policy to actively identify model prediction failures in the real environment and adds the resulting real-world trajectories to a repair pool. The fidelity gate temporarily excludes unreliable starting points from policy training and routes them to the repair pool. Once they pass subsequent checks, these starting points can be admitted into the policy curriculum.

Critical strengths
  1. The paper addresses a meaningful problem: when faced with a difficult experience, should the agent use it for policy learning or prioritize repairing the world model? The authors propose a curriculum-based routing mechanism, and the main experiments support its practical value.
  2. The experimental evaluation is extensive. The proposed method achieves improvements of 20–40% on challenging MQE tasks, and the ablation studies demonstrate the effectiveness of individual components.
  3. The paper is well organized and clearly written, with a logical structure and clear cross-references. The content is easy to follow.
Critical weaknesses
  1. Simply combining the two curricula does not seem particularly novel, but the fidelity gate adds an important element to the method. A sample may be rejected by the gate for different reasons, and the paper discusses two possible sources of performance improvement: (1) the policy avoids learning from erroneous imagined trajectories, and (2) the world model repairs regions that were previously unreliable but potentially useful for policy learning, allowing the policy to learn from these regions later. However, the main ablation studies do not separate these two effects. Although the appendix includes related comparisons, these experiments mainly examine whether samples should be filtered based on model fidelity. I would like to see an additional experiment addressing the following question: once unreliable starting points have already been excluded from policy training, does actively sending problematic regions identified by the policy curriculum for model repair provide additional benefits over standard world model training?

  2. The SMACv2 experiments match the number of task-policy interactions and updates, but the developer policy and its associated random policy require approximately one to two times as many additional real-environment interactions. The computational cost of the full system is also roughly twice that of the backbone. The current results provide strong evidence for the benefit of the fidelity gate compared with ablations under matched costs, but they do not yet establish that the full method is more sample-efficient than the backbone when all real-environment interactions are counted. I suggest that the authors include a comparison based on cumulative real-environment interactions and clarify that the current sample-efficiency claims are based on matched task-policy interaction budgets.

  3. The text in Figure 1 is too small compared with the main text. I suggest that the authors increase the font size to improve readability.

Questions for authors

Why does initializing the developer policy randomly lead to a higher task-policy success rate, as shown in the ablation study in Appendix C?

2 comments
AuthorsOpenReview ↗

Follow-up on supporting evidence and clarifications

Dear Reviewer 3pww,

Thank you again for your constructive feedback. As the discussion period draws to a close, we would like to highlight how our response addresses the points you raised:

  • Filtering and targeted repair: We provided supporting evidence from a separate six-scenario, three-seed comparison, where developer-driven collection and prioritized repair reduced held-out reward prediction error by 36.9–54.6%. We also clarified the evidence for quarantined states passing subsequent audits and regaining eligibility for prioritized training. These findings support the repair mechanism, while the incremental win-rate benefit over filtering alone remains unmeasured.
  • Total-interaction analysis: Following your suggestion, we replotted Terran 5v5, 10v10, and 10v11 counting all real-interaction streams. Our rebuttal explicitly limits the efficiency claim to higher task performance at matched task-policy interaction budgets, rather than greater efficiency over total interactions.
  • Developer initialization: We clarified the observed 45% versus 40% win rates for random versus task-actor initialization, and explained how the frozen KL reference could influence discovery of complementary failures. We distinguish this hypothesis from the measured performance difference and acknowledge that initialization and reference policy change together.

We will also improve Figure 1’s readability in the manuscript revision.

We hope these analyses and clarifications help resolve your concerns. Could you let us know whether any points remain that you would like us to address before the discussion closes?

Best regards,

The Authors

Thanks to the author for his reply and additional analysis. The comparison of the total interaction volume and the narrowing of the sample efficiency conclusion responded to my related doubts, and the explanation of developer initialization was also clearer.

Overall, the responses allay my main concerns, and the authors clearly acknowledge the remaining limitations of the evidence. I will maintain the original score (weak acceptance) and hope that the revised manuscript accurately presents these limitations and implements the modifications to the efficiency statement and illustrations.

Reviewer_pgSvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PROWL-2, a dual-curriculum framework for multi-agent model-based reinforcement learning that couples an imagined task-policy curriculum with active world-model repair via an environment-grounded fidelity gate. Building on IMAC (prioritizing imagined rollouts by positive value loss) and PROWL-1 (using a developer policy to find and repair world-model errors), the authors show that naively prioritizing high-TD-error imagined rollouts degrades performance because the curriculum disproportionately selects states where the world model is inaccurate. To resolve this ambiguity, PROWL-2 periodically audits high-priority start states by rolling the world model forward open-loop under the recorded real actions stored in the replay buffer and comparing predictions against the recorded continuation. States whose prediction error exceeds a reference threshold are quarantined from policy training and sent to a repair buffer (alongside trajectories from a developer policy) until subsequent audits confirm that world-model fidelity is restored. Experiments on nine SMACv2 scenarios (built on MARIE) and four MQE quadruped tasks (built on SeqWM) show consistent gains over model-based and model-free baselines.

Critical strengths
  1. The core insight of the paper—that prioritizing imagined rollouts by value error creates a harmful feedback loop unless gated by realized world-model fidelity—is important and convincingly demonstrated. The contrast between the ungated curriculum (Dual, which consistently underperforms the MARIE backbone across all nine SMACv2 scenarios in Table 1) and the gated curriculum (PROWL-2, which turns that degradation into substantial gains) provides clear empirical validation of this mechanism, further supported by the random and uncertainty gate controls in Appendix B.3.

  2. Auditing start-state fidelity by rolling the world model open-loop under the already recorded actions in the replay buffer is a practical idea that requires no extra environment interaction for the audit itself and adds less than 1% compute overhead.

Critical weaknesses
  1. The necessity of the expensive developer policy (\pi_D) is not isolated from the fidelity gate. On SMACv2, running and training the developer policy adds 1x to 2x additional real-environment interactions (which are not charged to the 1M task-step budget in Table 1 and Figure 2) and doubles the wall-clock GPU cost from ~100 to ~200 B200 GPU-hours per run (Table 7). Yet in Tables 1 and 2, the Repair ablation (which pays this full developer cost) yields only minor improvements over the backbone, while adding the lightweight fidelity gate (<1% compute) on top of the task curriculum accounts for almost the entire performance gain. Because the ablations add components cumulatively (Backbone -> Repair -> Dual -> PROWL-2), the paper never evaluates the most natural and sample-efficient variant: applying the task curriculum and the fidelity gate (and repairing the world model only on quarantined task-policy segments from D_env) without any developer policy. If gating and repairing on the task policy's own replay buffer already suffices, the extra 1x–2x environment budget and 2x compute spent on the developer may be largely redundant.

  2. The instantiations on SMACv2 and MQE differ substantially in their core mechanisms, and the SMACv2 fidelity score relies on hand-crafted, game-specific feature parsing. On SMACv2, thefidelity metric (Appendix A.1) requires manually indexing specific health channels per race (6 features for Terran/Zerg, 7 for Protoss, with a 0.05 death threshold) and checking game-specific action-mask rules. Meanwhile, on MQE (Table 6 and Appendix D), PROWL-2 does not prioritize imagined rollouts (it reduces to standard TD-error prioritized replay over real transitions), does not train a separate developer policy, and reports training-episode success rates with exploration noise active rather than held-out evaluation episodes. In addition, Appendix E notes that SeqWM's warm-up on MQE was reduced from the official 50K steps (with 2,500 pre-training updates) to 2,000 steps with no pre-training, which may handicap the standalone SeqWM baseline before the fidelity gate is applied.

Questions for authors
  1. How does PROWL-2 perform if you remove the developer policy completely (so zero extra developer environment steps are collected, and the world model is repaired only on the quarantined high-priority segments flagged by the fidelity gate from the task policy's own replay buffer)?

  2. On SMACv2, how does the sample-efficiency comparison in Figure 2 look if the x-axis plots total real environment steps (including the 1M–2M steps collected by the developer policy and its matched random rollouts)? Also, how well does the SMACv2 fidelity gate work if it uses only generic observation and reward prediction errors (E_obs and E_r) without the hand-crafted ally-death and action-mask terms?

  3. On MQE (Appendix E), why was SeqWM's warm-up reduced from 50K steps and 2,500 world-model pre-training updates to 2,000 steps with no pre-training, and why are training-episode success rates (with exploration noise enabled) reported in Table 2 instead of separate evaluation episodes without exploration noise?

1 comment
AuthorsOpenReview ↗

Follow-up on additional results and clarifications

Dear Reviewer pgSv,

Thank you again for your constructive feedback. Your questions helped us strengthen the empirical evidence and clarify the scope of our claims. As the discussion period draws to a close, we would like to highlight the following updates:

  • Completed developer-off experiment: The requested ablation has now reached 1M task-policy steps across three seeds on both Terran 5v5 and Protoss 5v5. Full PROWL-2 exceeds developer-off by 6.2 and 4.1 percentage points, respectively. These results support the contribution of developer-driven collection with the gate active at matched task-policy budgets, while acknowledging its additional interaction cost.
  • Total-interaction analysis: We replotted all three Terran scenarios counting task-policy, developer, and matched random-policy interactions. Based on this analysis, we have explicitly limited our claim in the rebuttal to higher task performance at matched task-policy interaction budgets, rather than greater total-interaction efficiency.
  • MQE clarification: We verified the shared 10K-interaction initialization, clarified that the reported success rates measure online training performance with exploration enabled, and explained the implementation differences between MQE and SMACv2.

We hope these additional results and clarifications address your main concerns. Could you let us know whether any questions remain, particularly regarding the completed developer-off comparison? We would appreciate the opportunity to address them before the discussion closes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.