acceptodds
Under review as a conference paper at ICLR 2027

Robust Planning in Stochastic Environments via Selective Adversarial Interventions

Abstract

Stochastic MuZero and Stochastic AlphaZero have achieved the state-of-the-art for planning in stochastic environments by modeling stochasticity using afterstates and chance events. Their expectation-based training focuses on maximizing average outcomes, at the expense of robustness to rare but consequential chance events. To address this, we introduce Robust Stochastic Zero (RSZ), the first method to extend Stochastic AlphaZero using robust adversarial reinforcement learning (RARL) with selective adversarial interventions. Unlike typical RARL methods that constantly perturb states, RSZ intervenes only at the most critical afterstates, which rarely occur but cause significant value degradation. Furthermore, it leaves the otherwise dynamics untouched, minimizing the interventions to avoid distorting the inherent stochastic dynamics. This approach improves the planning robustness without obstructing the ability to plan in the natural environment. On benchmark stochastic environments, 2048, Tetris Block Puzzle, and Server Allocation, RSZ achieves improved performance over the baseline with intervening in only 0.01% of events, and remains comparable when no interventions are applied. Our findings demonstrate that the right rather than constant interventions are a direction for robust planning in stochastic environments.

Primary area:
reinforcement learning
Keywords:
AlphaZero, robust adversarial reinforcement learning, stochastic environments, search-based planning, MCTS

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_CtuFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Robust Stochastic Zero (RSZ), which extends Stochastic AlphaZero with selective adversarial interventions at critical afterstates. It formulates a criticality score including severity and rarity score on afterstate to determine where the intervention occur while modified MCTS backups to emphasize adverse outcomes. Empirical studies on various stochastic planning games support majority of author's claims.

Critical strengths

1, This paper seems address an important and interesting problem, robustness to rare but consequential chance events, existing in the expectation-based planning algorithm.
2, By using 2048 case study, author gives a pretty illustrative example on importance of finding critical afterstate, and how and why RSZ's intervention on afterstate works without effecting planning in other scenarios.
3, The definition of criticality is helpful as a metric to define critical afterstate and corresponding adversarial training procedure. \

Critical weaknesses

1, The proposed criticality is clearly defined, however, it is still unclear for reviewer to understand how reliably the score could identify the critical situation to intervene, given the estimated values may be inaccurate during training.
2, RSZ requires roughly twice the computation power/resource comparing with the baseline, and even though the paper includes useful ompute-matched comparisons in the empirical studies, the results on the main environments are mixed leaving the practical trade-off somewhat unclear.
3, Empirical staudies mainly uses adversaries built around the proposed methods, and it would be helpful to test whether the benefits extend to other types of unexpected events or adversarial strategies.

Questions for authors

1, What is the intuition or trade-off in choosing the \lambda portion of criticality for training?
2, Also is the intervention set fixed, policy-dependent, or history-dependent in the optimization problem defined?
3, Under what conditions do the modified backups converge to the Bellman minimum at critical afterstates and the nominal expectation elsewhere, particularly when nodes switch classifications?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CtuF

Dear Reviewer CtuF

We sincerely appreciate your valuable feedback. We hope our clarifications address your concerns and strengthen the contributions of our work.

Response to questions:

1, What is the intuition or trade-off in choosing the \lambda portion of criticality for training?

The intuition is to let the adversary focus on afterstates that can lead to severe consequences that are rarely encountered naturally. Criticality is designed to combine severity and rarity to efficiently identify such afterstates. A small λ\lambda is used to avoid distorting the environment too much. Given the current state, action aa deterministically leads to afterstate asas. An adversarial intervention at asas can therefore help the agent learn the risk associated with aa.

The trade-off is between exposure to unfortunate events and distortion of the original environment. A smaller λ\lambda concentrates interventions on the most critical afterstates but may provide insufficient training exposure; a larger λ\lambda broadens exposure but can alter the training distribution excessively.

2, Also is the intervention set fixed, policy-dependent, or history-dependent in the optimization problem defined?

The intervention set is not fixed during training. Specifically, the agent's evolving policy changes its chosen actions and encountered afterstates, which changes the criticality distribution and the threshold for targeting its top λ\lambda portion. Thus, the intervention set is policy-dependent and history-dependent through the online estimator (Section 4.2 and Appendix A.4).

With λ=0.01\lambda=0.01%, the estimator approximately targets the highest-criticality 0.01% of recently encountered afterstates. Their identities and the threshold can change as the policy and value estimates evolve.

3, Under what conditions do the modified backups converge to the Bellman minimum at critical afterstates and the nominal expectation elsewhere, particularly when nodes switch classifications?

Thank you for raising this theoretical question. Here, we assume minimum and expectation refer to local backups: minimizing over chance-event returns at critical afterstates, and averaging them using the original chance probabilities elsewhere.

A possible convergence argument follows a UCT-style induction (Kocsis & Szepesvári, 2006): starting from exact terminal values, establishing convergence of parent values from convergent child values, and proceeding to the root. As child values converge, criticality measures can converge as well. When their limits lie strictly above or below the threshold, classifications eventually stop switching. PUCT would then increasingly favor worst events at critical nodes, while QRS would follow the original chance probabilities elsewhere, yielding the corresponding local backups.

This argument assumes a finite tree, bounded values, exact terminal evaluations, fixed threshold, fixed network predictions, a unique worst event at each afterstate, and infinitely many visits to every relevant child. These are conditions for a potential proof; a formal convergence analysis remains future work, as discussed in Section 6.

Response to other weaknesses:

it is still unclear for reviewer to understand how reliably the score could identify the critical situation to intervene, given the estimated values may be inaccurate during training.

Inaccurate value estimates can lead to incorrect intervention decisions, particularly early in training. However, value estimates can improve during training, as observed in AlphaGo Zero (Silver et al., 2017, Figure 3c), potentially making criticality more reliable. Even with inaccurate values, interventions may expose the agent to adverse events, although benefits are not guaranteed. We have added Appendix A.5 to discuss these effects.

the results on the main environments are mixed leaving the practical trade-off somewhat unclear.

Thank you for pointing out this unclear practical trade-off. We agree that RSZ is not always more compute-efficient than SAZ. Nevertheless, its additional cost comes from adversarial search during training; deployment does not require searching on the environment side. Therefore, RSZ is more compelling when robustness to rare but severe chance events is important and additional offline training is affordable. We have made this practical trade-off more explicit in Section 6, and discussed a hybrid future direction that begins with SAZ for efficient initial learning and gradually introduces RSZ's adversarial training.

it would be helpful to test whether the benefits extend to other types of unexpected events or adversarial strategies.

We agree that additional evaluations using other adversarial strategies, such as those in our ablation study, would help assess robustness beyond criticality-based attacks. However, these evaluations are infeasible within the rebuttal time and computational budget. They remain future work.

Reviewer_CtuFOpenReview ↗

Reply

Thanks for the author's effort in addressing my questions. I will maintain my score.

Thank you again for your constructive feedback and for considering our responses.

Reviewer_Yh9sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Robust Stochastic Zero (RSZ), an extension of Stochastic Alpha Zero using selective adversarial interventions to minimize disruptions to nominal planning while reducing tail risk. RSZ measures a criticality metric which, above a threshold, triggers an adversarially-chosen perturbation to minimize the planner's performance. The paper demonstrates that training under this rare-event regime improves the robustness of the final policy.

Critical strengths
  • the idea is sensible in that a planning model should preserve nominal dynamics, instead of only focus on worst case.
  • the example figures, while somewhat large, are helpful.
Critical weaknesses
  • The related works mentions several other tail risk-related competing baselines (RobustZero, RiskZero), but only evaluates stochastic alphazero as a baseline. A comparison to the more recent literature woud significantly improve the paper's argument.
  • The empirical setup is circular at times. Specifically, the methods are evaluated against RSZ's trained adversaries, so RSZ is evaluated in-distribution; also showing criticality in Table 3 as a measure of success is somewhat trivial as RSZ is specifically trained to minimize criticality.
  • λ=0.01\lambda =0.01% should be more clearly motivated or justified as a training parameter. A sensitivity analysis may explain the high variance in outcomes of RSZ across testing values of λ\lambda seen in Section 5.4
  • The paper does not discuss the limitations of the method
Questions for authors
  • Could the authors expand on the decision to use λ=0.01\lambda=0.01% in training?
  • What circumstances would limit the effectiveness of RSZ?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Yh9s

Dear Reviewer Yh9s

We sincerely appreciate your valuable feedback. We hope our clarifications address your concerns and strengthen the contributions of our work.

Response to questions:

Could the authors expand on the decision to use λ=0.01\lambda = 0.01 in training?

λ=0.01\lambda = 0.01 should be more clearly motivated or justified as a training parameter.

Thank you for pointing out the lack of explanation. We have improved Section 6 to better clarify this decision. Our training intervention rate λ=0.01\lambda = 0.01% is chosen as a conservative, small λ\lambda intended to limit environmental distortion. Higher λ\lambda may substantially change the event distribution and adversely affect performance, while the optimal λ\lambda depends on the environment, as demonstrated in Figure 13. In practice, we therefore recommend starting with a small λ\lambda and exploring larger ones when additional computation is available.

What circumstances would limit the effectiveness of RSZ?

The paper does not discuss the limitations of the method

RSZ's effectiveness can be limited in several circumstances.

  1. Adversarial planning adds training cost. Under a tight computational budget, allocating this computation to additional SAZ training can be more beneficial. Appendix H.1 illustrates this trade-off.
  2. Identifying critical afterstates depends on learned estimates and explored chance events. Inaccurate estimates or an insufficient search may cause RSZ to miss intervention opportunities. Appendices A.1, A.3, and A.5 describe these mechanisms.
  3. The intervention rate is environment-dependent. Too few interventions may provide insufficient exposure, while too many can distort training. Appendix G demonstrates this sensitivity.
  4. The current implementation also assumes access to a simulator and chance-event probabilities through SAZ, and our evaluation focuses on single-agent environments with discrete chance events. Extending beyond these assumptions requires further study.

We have expanded Section 6 to make these practical limitations explicit and connect them to future directions.

Response to other weaknesses:

The related works mentions several other tail risk-related competing baselines (RobustZero, RiskZero), but only evaluates stochastic alphazero as a baseline.

RobustZero and RiskZero assume deterministic transitions. SMZ's Figures 2 and 3 (left subfigures) demonstrate substantial limitations of deterministic models in stochastic environments, indicating that directly applying these methods to our setting is inappropriate. We therefore use SAZ as the stochastic planning baseline and compare alternative approaches within the same framework in Table 2. We have expanded this explanation in Appendix I.

The empirical setup is circular at times.

Each RSZ agent is evaluated against multiple adversaries, only one of which was jointly trained with it. Appendix F.2 (Tables 7-9) now provides complete pairing results and analysis. Excluding jointly trained pairs, RSZ retains higher aggregate mean return and CVaR at 10% than SAZ in all three environments. Performance against jointly trained adversaries shows no consistent specialization pattern.

Table 3 as a measure of success is somewhat trivial as RSZ is specifically trained to minimize criticality.

We agree that lower criticality for RSZ is somewhat trivial. Section 5.3 is intended as supplementary behavioral analyses to check whether RSZ exhibits the intended behavior of reducing its exposure to critical afterstates. The main performance evaluations of returns and robustness are provided in Section 5.1.

Besides the trivial results, Table 3 also reveals where this effect is limited. In Tetris Block Puzzle, the criticality changes only from 0.7858 to 0.7777. One possible explanation is that λ=0.01\lambda = 0.01% provides insufficient exposure. We have revised Section 5.3 to mention insufficient exposure as a possible explanation.

Reviewer_pkWgOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Robust Stochastic Zero (RSZ), an extension of Stochastic AlphaZero (SAZ) that employs a measure of criticality to determine when an adversarial agent should strategically intervene during an episode. This approach aims to identify extremely rare but critical scenarios that could lead to catastrophic outcomes. The paper formalises the stochasticity of an environment by using afterstates and chance events, and implements a novel robust stochastic Monte Carlo tree search (RS-MCTS) methodology performed on both the states and afterstates. The paper combines ideas from planning and robust adversarial reinforcement learning. Experimental results are shown on 2048, Tetris Block Puzzle and Server Allocation environments. The main results show evaluation on the natural environments without adversarial intervention, and on the environments with intervention where trained adversaries are employed. The paper provides further experimental results on ablations of alternative intervention methods, the behaviour of the proposed criticality metric and the impact of varying intervention rates on the stochasticity of the environment.

Critical strengths
  • S1: The authors propose a substantial and justifiable extension to the SAZ architecture, and provide a clear and comprehensive methodology for its application and configuration.
  • S2: The detailed analysis of interventions and criticality during training provides a strong foundation for future research on stochastic environments involving rare catastrophic events.
Critical weaknesses
  • W1: The literature review highlights other similar extensions such as RobustZero(Li et al., 2025) to MuZero and AlphaZero. The extension are said to be a different setting to the ones given in the paper, but limited explanation is provided. Furthermore, in the alternative approach section of the results, none are included. Later in the Risk-averse RL of the Related Work (Chow et al., 2018a; Grennbergy etal., 2022) are said to have been ‘…conducted in small-scale settings with shorter episode lengths.’ As these are introduced it is reasonable to either expect experimental results demonstrating their failure, or a more detail description of why the method does not translate.
  • W2: The empirical results are slightly misleading, and the separability of performance is not completely clear given limited training runs performed.
    • Firstly, in the original (Antonoglou, 2022) Backgammon is an additional experiment setting used for a stochastic domain, it is not mentioned why this is not used here but 2048 is.
    • Whilst the removal of tail model can be justified by the high variance of the results, with so few models trained, it does not feel appropriate here.
    • The evaluation of models in environments with an adversary is not entirely clear. As discussed later, these environments exhibit stochasticity that differs from that of the natural environments. Furthermore, the evaluation appears to assess the adversary and RSZ networks jointly, following their joint training. This raises concerns about the fairness of the comparison with SAZ, as RSZ will have been exposed during training to the altered environmental stochasticity induced by the trained adversary, whereas SAZ will not have had the same exposure.
    • Generally the results do not appear statistically separable. At one SD, Table 1 has many highly overlapping intervals. This does not provide reliable conclusions. This is also replicated less severely in Table 2
  • W3: The overarching focus of the paper is on improving robustness of the SAZ method. Whilst this is discussed implicitly, the paper would benefit from a small section to discuss precisely what the robustness is to, and why the results show evaluation on the adversarial environment. Appendix C.2 and K provides a good start to this in explaining the stochasticity of the natural environment.
  • W4: Whilst the explanation of the RSZ algorithm is sound and well presented, it is not immediately obvious to the reader what is paper contributions, and what is previous work.
  • W5: While Stochastic MuZero is proposed and formally defined in (Antonoglou, 2022), SAZ is not explicitly introduced or defined. Although the extension of the methodology to SAZ may be apparent to readers familiar with the field, the paper would benefit from a brief section explaining how SAZ is derived from the existing methodology.
Questions for authors
  • C1: The hyperparameters used for training are showed in Appendix D, but not comment is given on their selection. How are the hyperparameters chosen?
  • C2: The motivation for the minimal interventions is to not distort the underlying stochastics of the environment, however, analysis shows that for reasonably small intervention rates, the dynamics do vary. How do the authors justify their claim of not distorting the original stochastics of their method, particularly when the distorted environments are used in the evaluation?
  • C3: robust stochastic Monte Carlo tree search (RS-MCTS) is presented as being a novel contribution, however it is not clearly stated as such in the earlier text. Can the authors provide additional clarification if RS-MCTS is a novel contribution, or produced from previous work?
  • C4: L312 introduces zz as a training objective, but it is not defined or explained. Can authors provide clarification on its definition?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pkWg (1/2)

Dear Reviewer pkWg

We sincerely appreciate your valuable feedback. We hope our clarifications address your concerns and strengthen the contributions of our work.

Response to questions:

C1: The hyperparameters used for training are showed in Appendix D, but not comment is given on their selection. How are the hyperparameters chosen?

The hyperparameters are mostly based on related works, with modifications for our tasks and computational budget. We do not run an extensive hyperparameter search. We have expanded Appendix D.2 to include more details for the hyperparameters chosen.

C2: The motivation for the minimal interventions is to not distort the underlying stochastics of the environment, however, analysis shows that for reasonably small intervention rates, the dynamics do vary. How do the authors justify their claim of not distorting the original stochastics of their method, particularly when the distorted environments are used in the evaluation?

Thank you for pointing this out. We agree that our wording was too strong and have softened related statements. Our goal is to preserve the environment's original stochasticity as much as possible. However, adversarial interventions can still slightly change the distribution of chance events, even when they are sparse.

C3: robust stochastic Monte Carlo tree search (RS-MCTS) is presented as being a novel contribution, however it is not clearly stated as such in the earlier text. Can the authors provide additional clarification if RS-MCTS is a novel contribution, or produced from previous work?

RS-MCTS is a novel contribution proposed as the core of RSZ. In the abstract, to keep it concise, we still omit it. In the introduction, we have replaced the generic phrase "enhance Monte Carlo tree search" with a direct introduction of RS-MCTS.

C4: L312 introduces zz as a training objective, but it is not defined or explained. Can authors provide clarification on its definition?

Thank you for pointing out the missing definition. Following Stochastic MuZero, zz denotes the nn-step bootstrapped return used as the value target. We have explicitly defined it to remove this ambiguity in Section 4.4.

Response to other weaknesses (1/2):

The extension are said to be a different setting to the ones given in the paper, but limited explanation is provided. Furthermore, in the alternative approach section of the results, none are included.

RobustZero and RiskZero assume deterministic transitions. SMZ's Figures 2 and 3 (left subfigures) demonstrate substantial limitations of deterministic models in stochastic environments, indicating that directly applying these methods to our setting is inappropriate. We therefore use SAZ as the stochastic planning baseline and compare alternative approaches within the same framework in Table 2. We have expanded this explanation in Appendix I.

Later in the Risk-averse RL of the Related Work ... As these are introduced it is reasonable to either expect experimental results demonstrating their failure, or a more detail description of why the method does not translate.

Thank you for highlighting this insufficient explanation. Some risk-averse related methods could be applied to our environments. However, without tree search, the direct comparison cannot isolate the effectiveness of robust planning from the underlying planner. As SMZ's Figures 2 and 3 (right subfigures) demonstrated, changing the search budget significantly affected results. We have revised Section 2 and Appendix I to clarify this consideration.

Backgammon is an additional experiment setting used for a stochastic domain, it is not mentioned why this is not used here but 2048 is.

We focus on single-agent stochastic environments, where the adversary has a clear objective to challenge the agent. Backgammon is a two-player game, making the adversary's role less straightforward, as an intervention that disadvantages one player may benefit the other. We have revised Section 6 to clarify this limitation and identify it as a future direction.

Whilst the removal of tail model ... with so few models trained, it does not feel appropriate here.

For the main performance comparison in Table 1, we report the results of five trained models in Table 6 in Appendix F.1. These results generally follow the same trends as Table 1, but show substantially larger standard deviations. We retain the trimmed results in the main text for consistency with the subsequent experiments in Section 5. We have referenced the full results of five models from Section 5.1 so that readers can readily assess the full results and variability across training runs.

AuthorsOpenReview ↗

Response to Reviewer pkWg (2/2)

Response to other weaknesses (2/2):

This raises concerns about the fairness of the comparison with SAZ, as RSZ will have been exposed during training to the altered environmental stochasticity induced by the trained adversary, whereas SAZ will not have had the same exposure.

Thank you for raising this concern. We agree that RSZ receives adversarial exposure during training. This comparison assesses the overall benefit rather than isolating the effect of the interventions. To assess that intervention selection mechanism, Table 2 compares alternative approaches.

On the other hand, each RSZ agent is evaluated against multiple adversaries, only one of which was jointly trained with it. Appendix F.2 (Tables 7-9) now provides complete pairing results and analysis. Excluding jointly trained pairs, RSZ retains higher aggregate mean return and CVaR at 10% than SAZ in all three environments. Performance against jointly trained adversaries shows no consistent specialization pattern.

Generally the results do not appear statistically separable.

We agree that the small number of training runs limits the strength of our conclusions, but substantially expanding the main experiments is infeasible within our computational budget. This is a common challenge in many AlphaZero/MuZero papers. Appendix H.2 provides a smaller-scale study with 15 independent training runs per method on 3x3 2048 (Table 11), providing more statistically reliable evidence of RSZ's robustness gains. We have revised Section 5.1 to acknowledge the statistical limitation and reference this supplementary study.

the paper would benefit from a small section to discuss precisely what the robustness is to, and why the results show evaluation on the adversarial environment.

Thank you for this suggestion. We have incorporated the discussion of environmental stochasticity from Appendix C.2 into the beginning of Section 5 and expanded it to clarify our robustness objective and evaluation policy.

it is not immediately obvious to the reader what is paper contributions

We have clarified the distinction between inherited components and our contributions in Section 4. Specifically, RSZ inherits SAZ's afterstate-based stochastic planning and policy/value learning, while introducing criticality-based adversarial interventions, RS-MCTS, and additional adversarial policy training.

the paper would benefit from a brief section explaining how SAZ is derived from the existing methodology

Thank you for pointing out this ambiguity. SAZ refers to the AlphaZero adaptation for stochastic environments described in their Appendix B, "AlphaZero Implementation". The original SMZ paper directly calls it AlphaZero rather than explicitly naming it Stochastic AlphaZero. We have added a footnote in Section 3 to identify this source and briefly explain the adaptation.

Thanks, I’m satisfied by the response and changes. My only query would be on the results in Appendix F.2. These show that some of the seeds used for training vary greatly in results, but thats fairly common in these things of domains, and may be need to be addressed in a follow-up.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

Dear all reviewers,

We sincerely appreciate your thoughtful comments and constructive feedback. We have revised the manuscript as follows, with changes highlighted in blue in the new version:

  • Added complete agent-adversary cross-evaluation tables and analysis in Appendix F.2. (Reviewers pkWg and Yh9s)
  • Expanded practical limitations and corresponding future directions in Section 6. (Reviewers pkWg, Yh9s, and CtuF)
  • Clarified prediction inaccuracies and qualified convergence claims in Appendices A.3 and A.5. (Reviewer CtuF)
  • Expanded discussion of the scope and applicability of comparison methods in Section 2 and Appendix I. (Reviewers pkWg and Yh9s)
  • Clarified the robustness objective, evaluation rationale, and claims about preserving stochasticity in Sections 1, 4, and 5. (Reviewer pkWg)
  • Added explanations of hyperparameter sources and selection in Appendix D.2. (Reviewer pkWg)
  • Clarified statistical limitations and references to existing supplementary results in Section 5.1. (Reviewer pkWg)
  • Improved discussion of the criticality analysis in Section 5.3. (Reviewer Yh9s)
  • Clarified the contributions, SAZ's source, and the definition of zz in Sections 1, 3, and 4. (Reviewer pkWg)

We have provided the complete response to you below your official review. Please let us know if you have any additional suggestions. Thank you for taking the time to review our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.