acceptodds
Under review as a conference paper at ICLR 2027

Compact Shield Representations via Safety-Preserving Decision-Tree Approximation

Abstract

Shielding is a well-established technique for enforcing formal safety guarantees in reinforcement learning. However, its practical deployment remains limited in part by the lack of interpretability of synthesized shields. Although recent work has explored decision trees (DTs) as explainable representations of shields, exact DT representations can become prohibitively large. We address this limitation by introducing an algorithm that learns compact DT approximations of safety shields by explicitly trading permissiveness for representation complexity. To preserve the safety guarantees of the original shield, the approximation never classifies an action deemed unsafe by the original shield as safe. To mitigate the resulting performance degradation, we introduce an approximation cost that accounts for the agent's state visitation frequencies and action preferences, together with the risk induced by each action. A user-defined threshold controls the trade-off between fidelity to the original shield and DT complexity. We evaluate our algorithm across diverse domains and show that allowing a small amount of conservative approximation can substantially reduce DT size while preserving the safety guarantees of the original shield and causing at most small changes in agent performance.

Primary area:
interpretability and explainable AI
Keywords:
Reinforcement Learning, Shielding, Decision Tree, Safe AI

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_Yu16OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies shielding in safe RL, which aims to improve the interpretability of synthesized shields. The authors address the limitation of decision trees (DT) by learns its compact representation of safety shields. The authors propose an algorithm to to maintain the safety guarantees and to prevent performance deterioration of the original shield. The authors evaluate our algorithm across diverse domains and show that allowing a small amount of conservative approximation can substantially reduce DT size.

Critical strengths
  1. I think the problem setting studied in this paper is important and interesting. The motivation is clearly presented and I found this paper is easy to follow.
  2. The proposed algorithm is technically sound and I think it is easy to implement.
  3. The source code is attached and I feel it is easy to reproduce (though I did not run it on my env).
  4. The empirical evaluation has been conducted in various domains. The results seem good to me.
Critical weaknesses
  1. It is unclear how original (complex) DTs are made compact by the proposed algorithm. After all, the compression rate largely depend on how verbose the original rule is.

  2. I wondering if the importance of this paper might decrease because we can use LLMs to interpret complicated rules.

  3. I also wonder to what extent there are situations where we want to simplify the DT even at the sacrifice of performance or safety?

Questions for authors

I would like the authors to see the Critical Weaknesses and provide their responses.

4 comments
AuthorsOpenReview ↗

Question 1 and 2

We sincerely thank the reviewer for their thoughtful feedback. We appreciate their positive assessment of the problem setting, technical soundness, reproducibility, and empirical evaluation of our approach. We also value their insightful questions regarding compression effectiveness, the role of LLMs, and the trade-offs between interpretability, performance, and safety, which we will carefully address in our response.

W1 How are the DTs made compact?

Shields map states to sets of safe actions, specifying which actions an RL agent is permitted to execute in a given state. While a shield can be represented exactly as a DT, such an exact representation often results in large trees.

Our approach addresses this challenge by constructing more compact DT representations through controlled approximations. Specifically, instead of preserving the exact set of safe actions in every state, our algorithm allows certain leaves to contain a subset of the originally permitted safe actions. This enables multiple regions of the state space to be merged into a single leaf, thereby reducing the overall tree size. Importantly, this approximation is conservative: it may restrict additional actions but never introduces unsafe ones, thus preserving the safety guarantees of the original shield.

However, restricting the available safe actions can potentially affect the RL agent's performance, particularly when actions preferred by the learned policy are excluded. To mitigate this effect, our algorithm takes into account the state visitation frequencies of the RL policy. It prioritizes preserving the original sets of safe actions in frequently visited states, where additional restrictions are more likely to influence the agent's behavior.

We ensure that the compression rate is not artificially inflated by computing it relative to the exact representation. In general, the achievable compression rate depends on the structure of the shield. For example, Pac-Man cannot be compressed as effectively as Museum Heist because precise planning is required to avoid the ghosts. Nevertheless, the compression rate was at least 50% across all examples.

W2 Importance in the age of LLMs

In fact, we see great potential in combining LLMs with shielding and explainable AI techniques, such as the DT learning approach proposed in this paper. These techniques complement each other in the following ways:

  • Safety shields are formally computed and provide, for each state-action pair, the exact probability of a future safety violation. However, they do not provide any human-interpretable explanations for their decisions.
  • DT learning extracts interpretable predicates that relate the shield's decisions to properties of the environment. For example, a DT may learn that “driving fast is forbidden when a robot is less than one meter away from a human”, whereas the original shield merely lists all the states in which driving fast is forbidden.
  • Integrating LLMs could enable interactive, natural-language explanations of the shield's decisions at runtime. By following the paths of the DT, an LLM could explain to a human why a particular action is considered unsafe in the current situation.

Thus, combining formal shields, DT learning, and LLMs could represent a significant step toward safe and explainable learning-based autonomous systems. We are excited to explore this promising direction in future research.

AuthorsOpenReview ↗

Question 3

W3 Application scenarios

The motivation for compact shield representations is that shields should enhance trust in AI-based autonomous systems. However, as shields are currently mostly represented as huge lookup tables that map states to actions without providing any insights into their reasoning, they fail to enhance trust for two main reasons:

  • Both the decision-making processes of the shield and the AI system are opaque. Thus, even with a shield, users and developers have a difficult time testing, debugging, or understanding the actions of the shielded AI system.
  • Since the actions of the shield are not understandable, it is also unclear whether the shield is active and ensuring safety, or whether assumptions about the environment made in the safety analysis of the shield may have changed, rendering the shield inactive.

Therefore, we need an explainable representation of a shield to enhance trust in an AI system, i.e., to be able to understand why the actions of a shielded system are safe.

DT representations have proven to be a standard approach in explainable AI [1,2,3,4]. Current applications of shields are mostly in games, simulated environments, or robotics, where no critical interactions with humans take place [5]. Thus, even though shields are intended to be applied in the most safety-critical systems, they are not yet prominent in these areas.

We hope that by providing explainable representations of shields, we can advance the application of shields in critical autonomous systems and infrastructure, such as manufacturing, urban driving, aerospace engineering, water, energy, and healthcare systems.

Furthermore, please note that our algorithm preserves the safety guarantees of the original shield: Any action classified as unsafe by the original shield is also classified as unsafe in the approximate DT representation.

[1] Guidotti, Riccardo, et al. "A survey of methods for explaining black box models." ACM computing surveys (CSUR) 51.5 (2018): 1-42.

[2] Milani, Stephanie, et al. "Explainable reinforcement learning: A survey and comparative review." ACM Computing Surveys 56.7 (2024): 1-36.

[3] Rudin, Cynthia, et al. "Interpretable machine learning: Fundamental principles and 10 grand challenges." Statistic Surveys 16 (2022): 1-85.

[4] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." Advances in neural information processing systems 31 (2018).

[5] H Odriozola-Olalde, et al. “Shielded reinforcement learning for industrial applications: A systematic literature survey.” Intelligent Systems with Applications, 2026

Thank you for the responses. My concerns have been addressed. That being said, I do not think that this paper should be clearly accepted, so I will keep the original rating.

Thank you for your thoughtful review and for confirming that your concerns have been addressed. Your questions about LLMs and about when simplification is worth a trade-off pushed us to articulate the motivation and future directions more clearly, and we are glad the answers were convincing.

If any point remains open that we could still clarify or improve, we would be happy to do so. Thank you again for your time and constructive feedback.

Reviewer_NAduOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary

Using decision trees to interpret synthesised shields for safe reinforcement learning agents can become prohibitively large. The authors propose a method to learn compact DR approximations of the shield which trade off shield action-selection freedom for a much more compact tree.

Critical strengths

Strengths

  • Fig 1 is fantastic, it's inclusion helps the paper's readability and makes the methodology less dense.
  • The problem the paper tackles is an interesting take on the safe RL/shielding landscape, and has sensible framing throughout the paper.
  • The paper has a great breadth of benchmarks in the experimental evaluation, from toy examples like lunar lander to much harder tasks like the CARLA sim.
Critical weaknesses

Weaknesses

  • A few key metrics are absent in my opinion. I would like to see violation rates, goal-reaching (if possible), etc. I get the shield could be more restrictive but, it would be better to show this. Also reward could task performance, it would be beneficial to show task performance so a reader can empirically check the agent has not found someway to reward hack.
  • An ablation study of the split rule in the methodology would be beneficial. The methodology pciks which leaf to split by cost, and then proceeds to pick how to split it using entropy, which then ignores the cost. Could the authors compare against a cost-aware split rule, and clarify whether splitting can ever increase the tree's 'cost'?
  • Precision is a cost-weighted recall of permitted actions in the experimental evaluation, at first glance it may be interpreted as the fraction of safe actions that may be misclassified. Either changing the metric name or clarifying this in Section 6 would be great for readability.
  • The introduction sells the method has been fully evaluated on six benchmarks including Boeing and CARLA. This obviously piqued my interest, and I was wow-ed. Seeing that those two were only partly experimented on was a shame. Could the claims be tempered please.
  • Hyperparams for MaskablePPO are not given, or please point me to them?
  • Novelty is technically incremental in my opinion for this venue, I am quite fond of the paper but it feels similar to previous methods such as dtControl+ϵ\epsilon (your references line 564?). I am yet to be convinced about its full novelty.

Minor issues

  • Related work is quite short for the paper. For readability, the flow would make much more sense before the preliminaries as it sets up your methodology a lot better for readers unfamiliar with the topic.
  • Some key papers could also be included which are quite relevant to shielding/shielding representations/etc: [1] Rahman, M.A., Liu, T. and Alqahtani, S.M., 2023, August. Adversarial Behavior Exclusion for Safe Reinforcement Learning. In IJCAI (pp. 483-491). [2] Bethell, D., Gerasimou, S., Calinescu, R. and Imrie, C., 2025. Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding. In: Proceedings of the 28th European Conference on Artificial Intelligence (ECAI 2025) [3] Le, T., Shefin, R., Gupta, D., Le, T. and Alqahtani, S., 2026. AEGIS-RL: Abstract, Explainable Graphs for Integrated Safety in RL. Autonomous Agents and Multi-Agent Systems, 40(2), p.33.
Questions for authors

Questions

  • Can empty action sets occur in the approximate tree? How often did the authors come across this issue if so?
  • Following on from above, what does the agent do in these cases?
  • What evidence is there that these trees maintain interpretability after your method?
4 comments

We sincerely thank the reviewer for their insightful and thorough feedback. We would first like to address the questions and then discuss the identified weaknesses. We have carefully considered all questions and suggestions and have addressed each of them in the revised version of the paper. We have also conducted additional experiments and included their results in the appendix. Please refer to the revised version of the paper, where all changes and newly added material are highlighted in blue. Questions:

Q1: Can empty action sets occur in the approximate tree? How often did the authors come across this issue if so?

Yes, empty action sets can occur. We now explicitly clarify this in the paper (Lines 242- 244 and 297). There are two general types of shields:

  • Shields that provide absolute guarantees: These never produce empty action sets but tend to be overly restrictive.
  • Shields that provide probabilistic guarantees: These may produce empty action sets but generally allow for higher performance.

The shields we use belong to the second category. Consequently, the original shields already encounter empty action sets, and our approximate shields may increase the frequency of such occurrences.

We report the exact numbers now in Table 6; Appendix F. To summarize for the fine-tuned approximation (RQ2):

EnvironmentOriginal ShieldFine-tuned DT
Lunar Lander0.0%0.4%
Pac-Man0.2%1.5%
Windy City8.1%20.4%
Museum Heist19.3%35.5%

Note that fallback actions could, in principle, be incorporated into both the original shield and its decision-tree representation. However, we chose not to do so, as this is not standard practice in the shielding community. We implemented this approach as an extension and report the results in Appendix G.

Q2: Following on from above, what does the agent do in these cases?

If the leaf label is empty, the agent executes the fallback action, i.e., the safest currently available action, as proposed in [1].

[1] Bettina Könighofer, Roderick Bloem, Nils Jansen, Sebastian Junges, and Stefan Pranger. Shields for Safe Reinforcement Learning. Communications of the ACM, 68(11):80–90, 2025. ISSN 0001-0782. doi:10.1145/3715958.

Q3: What evidence is there that these trees maintain interpretability after your method?

We followed the general result of [2] that smaller trees are more interpretable and manually analyzed some of them. We also added two DTs to the appendix. All trees are also included in the supplementary material.

[2] Piltaver, Rok, Luštrek, Mitja, Matjaž Gams, and Sanda Martinčić-Ipšić. "What makes classification trees comprehensible?." Expert Systems with Applications 62 (2016): 333-346.

AuthorsOpenReview ↗

Weaknesses

W1 Additional metrics in the experimental evaluation, such as violation rates, goal-reaching rates, and task performance.

We agree with the reviewer that including these metrics is helpful. We have added a new Table 3 (Lines 443–469), reporting violation rates and goal-reaching rates where applicable. In summary, violations are rare and, as expected, occur at similar rates across all shield representations. Success rates may decrease slightly when using the approximate shield (e.g., by 1.6 percentage points at most for Pac-Man) and generally follow the same trends as the rewards.

W2 (a) Missing ablation study and comparison against cost-aware splitting rules

We performed a full ablation study (Appendix H) and added a short discussion to the revised version of the paper (Line 473-475).

For all problem instances, we compared entropy splitting with multi-label entropy splitting [3], cost-space splitting, as suggested by the reviewer.

To summarize for the policy-finetuned shields (RQ2):

EnvironmentDT size: EntropyDT size: Multi-labelDT size: CostReward: EntropyReward: Multi-labelReward: Cost
Windy City43.845.037.40.70.70.5
Museum Heist391.8517.8384.62.02.01.8
Lunar Lander21.023.418.2205.1170.339.8
Pac-Man445.4404.6449.441.041.041.0

As the ablation study shows, the splitting strategies yield similar DT sizes and average rewards. The only exception is the Lunar Lander example, where cost-based splitting reduces the average reward achieved with the resulting shields.

[3] Clare, Amanda, and Ross D. King. "Knowledge discovery in multi-label phenotype data." European conference on principles of data mining and knowledge discovery. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001.

W2 (b) Clarification, whether splitting can increase DT's costs.

No, costs cannot increase after splitting, as they depend on the intersections of available actions. After a split, all previously available actions remain available, while additional actions may become available, potentially reducing the cost.

We add a proof to Appendix H.

W3 Clarify the meaning of precision

We added a clarification in the paper (Line 412) stating that precision is the share of the total approximation cost that a DT retains, not the share of safe actions it retains.

W4 Overselling the CARLA and Boeing TaxiNet experiments

We thank the reviewer for spotting this and have fixed the claims. We have clarified this in the introduction of the updated version (Line 130-134): “We perform full evaluation on several RL environments like Lunar Lander (Towers et al., 2024) and Pac-Man (Gross et al., 2022) and further compute approximate DTs for the challenging domain of collision avoidance in the high-fidelity driving simulator CARLA (Dosovitskiy et al., 2017) and aircraft taxiing in the Boeing TaxiNet simulator (Fremont et al., 2020). RL training was not performed for the latter two.”

W5 Hyperparameters for MaskablePPO are missing

We thank the reviewer for pointing this out and now clearly state that we use the default parameters (Line 408) and Appendix D.

W6 Novelty compared to dtControl

The tool dtControl2+ε\varepsilon computes epsilon-close decision trees for optimal deterministic policies that map states to actions.

However, shields implement nondeterministic policies that map states to sets of actions (as they should restrict the RL agent as little as possible and leave the agent the freedom to explore any safe action). Therefore, the algorithms in dtControl2+ε\varepsilon (such as agency, dominance, ...) cannot be applied to computing shield representations.

While prior work has investigated the representation of non-deterministic policies as decision trees (DTs), e.g., [4], existing approaches typically focus on minimizing tree size by exploiting the flexibility afforded by non-determinism to construct smaller trees. As a result, much of the original non-determinism is typically lost in the DTs produced by these algorithms.

To the best of our knowledge, our approach is the first to introduce a DT learning algorithm that explicitly aims to preserve as much non-determinism as possible. This is particularly important in the context of shielding, where preserving non-determinism prevents the resulting shield from unnecessarily restricting the agent's available actions.

[4] Ashok, Pranav, et al. "dtControl 2.0: explainable strategy representation via decision tree learning steered by experts." International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Cham: Springer International Publishing, 2021.

MW1 Related work

We have added the requested citations and moved the related work to the front.

Thank you for this detailed discussion back, and thank you for your hard work during this rebuttal period. I wanted to check all of the additional content and the other reviewers concernscarefully and I am happy with it for the most part.

I am VERY familiar with the area and a lot of the related work. I would appreciate the chairs if the paper was accepted as it sits as a good contribution for the field. As a result I increase my score to reflect my appreciation for the paper.

All the best to ther authors.

Thank you for the thorough review and for the time you put into checking the revised material during the rebuttal period. Your feedback improved the paper, and we appreciate your kind words and your support very much.

Reviewer_G7BYOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes compact decision tree (DT) representations for RL safety shields. It fundamentally addresses an interpretability issue: exact but large DTs may be prohibitively complicate for human analysis. The paper therefore provides means for constructing an approximate safety mask based on a DT that is much smaller. The paper achieves this by introducing a weighting: A high cost weight for when the approximate DT removes an action that has high likelihood under the agent's policy, and a lower cost weight for when DT remove introduces and approximation error in state that has low probability under the agent' s state visitation distribution. Crucially, the approximate smaller DT shield is safe by construction, meaning that the approximate, smaller DT shield can never permit and action that is forbidden under the original shield. The method is evaluated on a number of discrete (?) environments, and achieves large reductions of the exact DT shield, without degrading the performance of the RL agent.

Critical strengths

A well motivated, nicely presented, and clearly articulated paper that addresses a relevant problem in a smart way. Reducing the size of the DT shield by factoring in state visitation as well as action likelihood is appropriate, and the reduction in DT size can be expected to improve interpretability of the shield. The fact that the approximate shield is at least as safe as the original shield is noteworthy. Potential performance degradation in terms of reward (but not safety incidents) is exactly what matters for safety critical domains.

Critical weaknesses

The is only a single baseline, comparing to other constrained RL methods would be interesting.

The paper does not report the actual costs incurred by the evaluated methods. I understand that the proposed methods is at least as safe as the exact DT, but seeing what that means in terms of empirical constraint violations, and how it compares to other baselines, would be interesting.

Moreover, as noted in the paper, small DTs may still be hard to interpret, and ideally the authors would evaluate empirically whether the DTs produces by their methods actually improve interpretability. E.g., a DT with 4k nodes is indeed one order of magnitude smaller than one sizes 40k, but may still not be human interpretable.

Questions for authors

Is the method limited to discrete state and action spaces?

Have you recorded the safety violations and/or could you report them alongside tables 1 and 2?

7 comments

We sincerely thank the reviewer for their constructive review, and we are glad that the reviewer finds the paper well motivated and the approach appropriate. We have carefully considered the identified questions and weaknesses and addressed them in the revised version of the paper. We also conducted additional experiments and added their results to the appendix. All changes and new material in the revised version are highlighted in blue.

Q1 Is the method limited to discrete state and action space?

No, our explanation method is not inherently limited to discrete state and action spaces. However, the probabilistic shielding framework considered in this paper requires a finite MDP abstraction. Thus, when applied to environments with continuous state and action spaces, both spaces must be discretized to construct the MDP on which the shield is computed.

Lunar Lander, which is included in our evaluation, is an example of an environment with continuous states. This discretization requirement is not specific to our explanation method but is inherited from probabilistic shielding, which relies on the availability of an MDP model. Since our method explains the decisions of a probabilistic shield, it inherits the same prerequisite.

To clarify this, we have added a sentence to the main text (Line 404), explicitly stating that Lunar Lander has continuous state spaces and that the shield is computed on a discrete abstraction.

Importantly, this limitation is specific to the class of probabilistic shields considered in our paper, rather than shielding in general. Probabilistic shielding is among the most established and widely studied shielding approaches for safe reinforcement learning [3]. Nevertheless, alternative shielding techniques have been developed for continuous and hybrid systems, including robust model predictive shielding [1] and shield synthesis for hybrid systems using UPPAAL [2]. As future work, we plan to investigate how to construct interpretable representations and explanations for shields obtained through these alternative approaches.

[1] S Li, O Bastani: “Robust model predictive shielding for safe reinforcement learning with stochastic dynamics”. IEEE International Conference on Robotics and Automation (ICRA), 7166-7172, 2020.

[2] AH Brorholt et al. “Uppaal Coshy: Automatic Synthesis of Compact Shields for Hybrid Systems. Reachability Problems. RP 2025. https://doi.org/10.1007/978-3-032-09524-4_7

[3] B Könighofer et al., “Shields for Safe Reinforcement Learning”. Communications of the ACM, 68(11):80–90, 2025. ISSN 0001-0782. doi:10.1145/3715958.

Q2 Have you recorded the safety violations and/or could you report them alongside tables 1 and 2?

We agree that this is an interesting addition to the paper and have introduced a new Table 3 (also contained in this response) and discussion (lines 460-469) into the evaluation. Detailed results are now also reported in Appendix E.

Violations and task success in % (means across 5 seeds). Columns 2–4: training violations; 5–8: evaluation violations; 9–12: evaluation success. T are the DTs from RQ1, TRL are the fine-tuned DTs from RQ2.

EnvironmentTrain Unsh.Train shieldTrain TViol. Unsh.Viol. shieldViol. TViol. TRLSucc. Unsh.Succ. shieldSucc. TSucc. TRL
Windy City36.421.622.90.020.018.616.80.080.081.483.0
Museum Heist100.00.40.2100.01.40.01.80.098.6100.098.2
Lunar Lander44.10.10.00.00.00.00.099.8100.0100.0100.0
Pac-Man97.41.01.33.80.40.80.60.099.497.898.4

To summarise the results: As expected under Theorem 1, violation rates for the original shield and its approximations are similar. They are also generally no higher than an unshielded agent (unless the unshielded agent does not successfully complete the task for Windy City). Success rates are also comparable.

AuthorsOpenReview ↗

Weaknesses

W1 Comparison to constrained RL methods

We agree with the reviewer that a direct comparison between shielded RL and constrained RL would be a very interesting study and, to the best of our knowledge, has not yet been conducted.

However, we deliberately did not compare shielded RL with constrained RL, since the focus of this paper is to make existing shielded RL approaches explainable. Thus, we evaluate whether computing more compact shield representations comes at the cost of representations that are too coarse and restrictive, which we find is not the case.

We agree that a comparison between shielded RL and constrained RL in general is a very interesting research question that should be addressed. However, we do not see how it fits into the scope of this paper. If the reviewer disagrees and has specific ideas in mind, we would be happy to run further experiments.

W2 Constraint violations

We understand this weakness as a build-up to the following question, which we have answered before and which led to an extension of our paper (Table 3 in the evaluation and Appendix E).

W3 Empirical interpretability

We agree that an empirical evaluation of the interpretability would strengthen the paper.

We have included two DTs generated by our code in Appendix I. The presented DTs are small and clearly show the states the agent is mostly occupying during its execution. To address the second part of the weakness, we agree that 4k nodes are still not as interpretable as one would like. Nevertheless, even in the case of Museum Heist, which we assume the reviewer refers to, we found that the DTs are generally reasonably balanced. The longest path in Museum Heist was 28 nodes for the exact shield representation and 19 for the fine-tuned one. As [4] found, while size is an important proxy, path length also plays a role.

However, we agree that DTs are often still difficult for humans to interpret. Thus, in future work, we plan to investigate combining shields, DT explanations, and LLMs. The idea is that an LLM could access the DT representation of the shield and explain to the user at runtime why the current situation is safe or unsafe. To do so, the LLM would only need to explain the predicates that hold along the current path in the tree. Consequently, the focus would shift from learning small DTs to learning DTs based on a small set of easily explainable predicates.

[4] Piltaver, Rok, Luštrek, Mitja, Matjaž Gams, and Sanda Martinčić-Ipšić. "What makes classification trees comprehensible?." Expert Systems with Applications 62 (2016): 333-346.

Thank you for your response and revision of the manuscript, I appreciate the clarifications and additional results.

I have some remaining confusion about the applicability to continuous MDPs. Above, you are stating that "our explanation method is not inherently limited to discrete state and action spaces" and at the same time that "when applied to environments with continuous state and action spaces, both spaces must be discretized". My read on this would be that your method is in fact only applicable to discrete MDPs... Granted, there are tricks that you can do, namely to discretize the state and action spaces, but this discretization step is not detailed in your paper as part of the method or evaluated, so it is unclear how it impacts performance and constrain violations. Thus, I feel like the limited applicability to discrete MDPs persists. This is relevant, because it affects the scope of your method.

Related to this, I am confused about the CARLA and TaxiNet environments. These seem to feature continuous state spaces? For clarity, I'd appreciate if the manuscript stated their exact state and action space! And how did you apply your method to these environments? Based on the same discretization as for LunarLander? Moreover, from the revised manuscript, what do you mean by "RL training was not performed for [CARLA and TaxiNet]". What did you do on these environments, then?

Regarding the violation results in Table 3, and Figure 16, thank you for adding those. They are close to what I wanted to see, but not exactly, since you are reporting the violation rate, defined as the "fraction of episodes in which the event of the safety property occur". This is an interesting metric, but it misses the magnitude of the violations. An episode where a single constrain violation occurs is counted the same as one where 100 violations occurs? Could you instead or additionally report episode return (so sum of rewards in one episode) over training time and episode violations (so sum of violations in one episode) over training time? This would reveal the reward and constraint trade off under the different methods at a more appropriate level. I understand that your method is primarily concerned with the interpretability of the shield, but since shields are typically used in safety critical domains, how your method affects constraint violations is crucial.

For now, I am keeping my score as is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.