- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Robust Stochastic Zero (RSZ), which extends Stochastic AlphaZero with selective adversarial interventions at critical afterstates. It formulates a criticality score including severity and rarity score on afterstate to determine where the intervention occur while modified MCTS backups to emphasize adverse outcomes. Empirical studies on various stochastic planning games support majority of author's claims.
Critical strengths
1, This paper seems address an important and interesting problem, robustness to rare but consequential chance events, existing in the expectation-based planning algorithm.
2, By using 2048 case study, author gives a pretty illustrative example on importance of finding critical afterstate, and how and why RSZ's intervention on afterstate works without effecting planning in other scenarios.
3, The definition of criticality is helpful as a metric to define critical afterstate and corresponding adversarial training procedure. \
Critical weaknesses
1, The proposed criticality is clearly defined, however, it is still unclear for reviewer to understand how reliably the score could identify the critical situation to intervene, given the estimated values may be inaccurate during training.
2, RSZ requires roughly twice the computation power/resource comparing with the baseline, and even though the paper includes useful ompute-matched comparisons in the empirical studies, the results on the main environments are mixed leaving the practical trade-off somewhat unclear.
3, Empirical staudies mainly uses adversaries built around the proposed methods, and it would be helpful to test whether the benefits extend to other types of unexpected events or adversarial strategies.
Questions for authors
1, What is the intuition or trade-off in choosing the \lambda portion of criticality for training?
2, Also is the intervention set fixed, policy-dependent, or history-dependent in the optimization problem defined?
3, Under what conditions do the modified backups converge to the Bellman minimum at critical afterstates and the nominal expectation elsewhere, particularly when nodes switch classifications?
3 comments
Reply
Thanks for the author's effort in addressing my questions. I will maintain my score.
Thank you again for your constructive feedback and for considering our responses.
Response to Reviewer CtuF
Dear Reviewer CtuF
We sincerely appreciate your valuable feedback. We hope our clarifications address your concerns and strengthen the contributions of our work.
Response to questions:
The intuition is to let the adversary focus on afterstates that can lead to severe consequences that are rarely encountered naturally. Criticality is designed to combine severity and rarity to efficiently identify such afterstates. A small λ is used to avoid distorting the environment too much. Given the current state, action a deterministically leads to afterstate as. An adversarial intervention at as can therefore help the agent learn the risk associated with a.
The trade-off is between exposure to unfortunate events and distortion of the original environment. A smaller λ concentrates interventions on the most critical afterstates but may provide insufficient training exposure; a larger λ broadens exposure but can alter the training distribution excessively.
The intervention set is not fixed during training. Specifically, the agent's evolving policy changes its chosen actions and encountered afterstates, which changes the criticality distribution and the threshold for targeting its top λ portion. Thus, the intervention set is policy-dependent and history-dependent through the online estimator (Section 4.2 and Appendix A.4).
With λ=0.01%, the estimator approximately targets the highest-criticality 0.01% of recently encountered afterstates. Their identities and the threshold can change as the policy and value estimates evolve.
Thank you for raising this theoretical question. Here, we assume minimum and expectation refer to local backups: minimizing over chance-event returns at critical afterstates, and averaging them using the original chance probabilities elsewhere.
A possible convergence argument follows a UCT-style induction (Kocsis & Szepesvári, 2006): starting from exact terminal values, establishing convergence of parent values from convergent child values, and proceeding to the root. As child values converge, criticality measures can converge as well. When their limits lie strictly above or below the threshold, classifications eventually stop switching. PUCT would then increasingly favor worst events at critical nodes, while QRS would follow the original chance probabilities elsewhere, yielding the corresponding local backups.
This argument assumes a finite tree, bounded values, exact terminal evaluations, fixed threshold, fixed network predictions, a unique worst event at each afterstate, and infinitely many visits to every relevant child. These are conditions for a potential proof; a formal convergence analysis remains future work, as discussed in Section 6.
Response to other weaknesses:
Inaccurate value estimates can lead to incorrect intervention decisions, particularly early in training. However, value estimates can improve during training, as observed in AlphaGo Zero (Silver et al., 2017, Figure 3c), potentially making criticality more reliable. Even with inaccurate values, interventions may expose the agent to adverse events, although benefits are not guaranteed. We have added Appendix A.5 to discuss these effects.
Thank you for pointing out this unclear practical trade-off. We agree that RSZ is not always more compute-efficient than SAZ. Nevertheless, its additional cost comes from adversarial search during training; deployment does not require searching on the environment side. Therefore, RSZ is more compelling when robustness to rare but severe chance events is important and additional offline training is affordable. We have made this practical trade-off more explicit in Section 6, and discussed a hybrid future direction that begins with SAZ for efficient initial learning and gradually introduces RSZ's adversarial training.
We agree that additional evaluations using other adversarial strategies, such as those in our ablation study, would help assess robustness beyond criticality-based attacks. However, these evaluations are infeasible within the rebuttal time and computational budget. They remain future work.