- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies shielding in safe RL, which aims to improve the interpretability of synthesized shields. The authors address the limitation of decision trees (DT) by learns its compact representation of safety shields. The authors propose an algorithm to to maintain the safety guarantees and to prevent performance deterioration of the original shield. The authors evaluate our algorithm across diverse domains and show that allowing a small amount of conservative approximation can substantially reduce DT size.
Critical strengths
- I think the problem setting studied in this paper is important and interesting. The motivation is clearly presented and I found this paper is easy to follow.
- The proposed algorithm is technically sound and I think it is easy to implement.
- The source code is attached and I feel it is easy to reproduce (though I did not run it on my env).
- The empirical evaluation has been conducted in various domains. The results seem good to me.
Critical weaknesses
-
It is unclear how original (complex) DTs are made compact by the proposed algorithm. After all, the compression rate largely depend on how verbose the original rule is.
-
I wondering if the importance of this paper might decrease because we can use LLMs to interpret complicated rules.
-
I also wonder to what extent there are situations where we want to simplify the DT even at the sacrifice of performance or safety?
Questions for authors
I would like the authors to see the Critical Weaknesses and provide their responses.
4 comments
Question 3
W3 Application scenarios
The motivation for compact shield representations is that shields should enhance trust in AI-based autonomous systems. However, as shields are currently mostly represented as huge lookup tables that map states to actions without providing any insights into their reasoning, they fail to enhance trust for two main reasons:
- Both the decision-making processes of the shield and the AI system are opaque. Thus, even with a shield, users and developers have a difficult time testing, debugging, or understanding the actions of the shielded AI system.
- Since the actions of the shield are not understandable, it is also unclear whether the shield is active and ensuring safety, or whether assumptions about the environment made in the safety analysis of the shield may have changed, rendering the shield inactive.
Therefore, we need an explainable representation of a shield to enhance trust in an AI system, i.e., to be able to understand why the actions of a shielded system are safe.
DT representations have proven to be a standard approach in explainable AI [1,2,3,4]. Current applications of shields are mostly in games, simulated environments, or robotics, where no critical interactions with humans take place [5]. Thus, even though shields are intended to be applied in the most safety-critical systems, they are not yet prominent in these areas.
We hope that by providing explainable representations of shields, we can advance the application of shields in critical autonomous systems and infrastructure, such as manufacturing, urban driving, aerospace engineering, water, energy, and healthcare systems.
Furthermore, please note that our algorithm preserves the safety guarantees of the original shield: Any action classified as unsafe by the original shield is also classified as unsafe in the approximate DT representation.
[1] Guidotti, Riccardo, et al. "A survey of methods for explaining black box models." ACM computing surveys (CSUR) 51.5 (2018): 1-42.
[2] Milani, Stephanie, et al. "Explainable reinforcement learning: A survey and comparative review." ACM Computing Surveys 56.7 (2024): 1-36.
[3] Rudin, Cynthia, et al. "Interpretable machine learning: Fundamental principles and 10 grand challenges." Statistic Surveys 16 (2022): 1-85.
[4] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." Advances in neural information processing systems 31 (2018).
[5] H Odriozola-Olalde, et al. “Shielded reinforcement learning for industrial applications: A systematic literature survey.” Intelligent Systems with Applications, 2026
Thank you for the responses. My concerns have been addressed. That being said, I do not think that this paper should be clearly accepted, so I will keep the original rating.
Thank you for your thoughtful review and for confirming that your concerns have been addressed. Your questions about LLMs and about when simplification is worth a trade-off pushed us to articulate the motivation and future directions more clearly, and we are glad the answers were convincing.
If any point remains open that we could still clarify or improve, we would be happy to do so. Thank you again for your time and constructive feedback.
Question 1 and 2
We sincerely thank the reviewer for their thoughtful feedback. We appreciate their positive assessment of the problem setting, technical soundness, reproducibility, and empirical evaluation of our approach. We also value their insightful questions regarding compression effectiveness, the role of LLMs, and the trade-offs between interpretability, performance, and safety, which we will carefully address in our response.
W1 How are the DTs made compact?
Shields map states to sets of safe actions, specifying which actions an RL agent is permitted to execute in a given state. While a shield can be represented exactly as a DT, such an exact representation often results in large trees.
Our approach addresses this challenge by constructing more compact DT representations through controlled approximations. Specifically, instead of preserving the exact set of safe actions in every state, our algorithm allows certain leaves to contain a subset of the originally permitted safe actions. This enables multiple regions of the state space to be merged into a single leaf, thereby reducing the overall tree size. Importantly, this approximation is conservative: it may restrict additional actions but never introduces unsafe ones, thus preserving the safety guarantees of the original shield.
However, restricting the available safe actions can potentially affect the RL agent's performance, particularly when actions preferred by the learned policy are excluded. To mitigate this effect, our algorithm takes into account the state visitation frequencies of the RL policy. It prioritizes preserving the original sets of safe actions in frequently visited states, where additional restrictions are more likely to influence the agent's behavior.
We ensure that the compression rate is not artificially inflated by computing it relative to the exact representation. In general, the achievable compression rate depends on the structure of the shield. For example, Pac-Man cannot be compressed as effectively as Museum Heist because precise planning is required to avoid the ghosts. Nevertheless, the compression rate was at least 50% across all examples.
W2 Importance in the age of LLMs
In fact, we see great potential in combining LLMs with shielding and explainable AI techniques, such as the DT learning approach proposed in this paper. These techniques complement each other in the following ways:
Thus, combining formal shields, DT learning, and LLMs could represent a significant step toward safe and explainable learning-based autonomous systems. We are excited to explore this promising direction in future research.