acceptodds
Under review as a conference paper at ICLR 2027

Selecting Decision-Relevant Concepts in Reinforcement Learning

Abstract

Training interpretable concept-based policies requires practitioners to select which human-understandable concepts an agent should use for sequential decision-making. This process requires domain expertise, scales poorly with the number of candidate concepts, and provides no performance guarantees. To overcome this limitation, we propose the first algorithms for principled automatic concept selection in sequential decision-making. Our key insight is that concept selection can be viewed through the lens of state abstraction: a concept is decision-relevant if removing it causes the agent to confuse states that require different actions. That is, states with the same selected concept representation should share the same optimal action, preserving the decision structure of the original state space. This perspective leads to Decision-Relevant Selection (DRS), which selects a subset of concepts from a candidate set and provides performance bounds relating abstraction error to policy performance. Empirically, DRS performs comparably to manually curated concept sets and improves policy performance under test-time concept intervention across reinforcement learning benchmarks and a healthcare simulator.

Primary area:
interpretability and explainable AI
Keywords:
concept, reinforcement learning, interpretability, state abstractions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PHJXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper selects a budgeted concept subset for RL by preserving distinctions between states with different action values. It proposes DRS and its noise-aware extension, DRS-log, with analysis based on state abstraction. Experiments cover five RL environments, simulated concept interventions, and supervised concept selection on CUB.

Critical strengths
  1. Useful formulation. Connecting concept selection to state abstraction provides a principled criterion for retaining decision-relevant information under a concept budget.

  2. Encouraging results. Reduced concept sets retain strong performance across several environments, with additional evaluation under imperfect predictions and simulated interventions.

Critical weaknesses
  1. The practical objective does not control the error used in the performance bound. Theorem 4.1 establishes minimax optimality with constraint (2d) and rho = 0. Practical DRS drops (2d), allowing its summed objective to trade a few large errors for many small improvements. The performance bound depends on the maximum error, which can remain large. Corollary 4.1 supplies no bound on the additional error from this relaxation, so it does not establish that practical DRS retains the idealized guarantee.

  2. Theorem 4.2 omits a condition on reference-policy quality. The statement bounds estimation error relative to QπQ^\pi, but the proof uses Q∗Q^*, effectively treating the reference policy as optimal. For a near-optimal reference policy, a bound such as ∥Qπ−Q∗∥∞≤η\|Q^\pi-Q^\ast\|_\infty\le\eta and a corresponding error term are needed. Without this link, accurate estimation of a suboptimal reference policy does not justify the stated performance guarantee. Appendix N provides empirical evidence but does not resolve this gap.

  3. DRS-log's analysis stops short of the claimed performance motivation. The analysis bounds pairwise separation probabilities under independent concept errors, but does not connect the optimized surrogate to maximum abstraction error or policy return. Consequently, it leaves unclear when accounting for predictor noise through DRS-log should preserve or improve decision performance.

  4. The interpretability benefit is underdeveloped. Selecting concepts using a reference policy and then training a new policy does not establish a faithful explanation of the reference policy. The new policy still uses a neural concept-to-action mapping. Reward and simulated correction results offer limited evidence of how this replacement helps a person understand, diagnose, or supervise decisions.

Questions for authors
  1. Is Theorem 4.2 intended to assume π=π∗\pi=\pi^*? If near-optimal reference policies are allowed, and how does the suboptimality enter the performance bound?

  2. How are Appendix L's maximum abstraction errors computed? Are they evaluated on held-out states, and do they quantify the error introduced by relaxing (2d)?

  3. What concrete inspection or diagnosis task benefits from replacing the reference policy with the new concept-based policy? Please illustrate with a specific state, concept values, and action or correction.

1 comment

Dear Reviewer PHJX,

Thank you for your review, and for viewing our paper as having a useful formulation with encouraging results. We provide answers to your questions below.

Theorem 4.2 omits a condition on reference-policy quality Is Theorem 4.2 intended to assume that π∗=π\pi*=\pi?

Thank you for your suggestion on Theorem 4.2. To deal with imperfections in the reference policy π^\hat{\pi}, we first define η:=max⁡s,a∣Qπ∗(s,a)−Qπ^(s,a)∣\eta := \max\limits_{s,a} |Q^{\pi*}(s,a) - Q^{\hat{\pi}}(s,a)|. Using the triangle inequality theorem, we can apply the same proof as Theorem 4.2, to demonstrate that $V^{{\pi*}_{\mathbf{c}}}(s) - V^{\pi*_{\hat{\mathbf{c}}}}(s) \le \frac{2(\epsilon_{Q^{\pi}}(g_{\mathbf c})+ 4\delta + 4 \eta)}{(1-\gamma)^2}$ where π∗c^\pi*_{\hat{\mathbf c}} denotes the optimal policy using the selected concepts. Here, c^\hat{\mathbf{c}} is selected using the approximate Q-values for the reference policy π^\hat{\pi}. The key intuition here is that a suboptimal policy is detrimental based on the degree that it reduces our knowledge of Q values.

The interpretability benefit is underdeveloped.

Thank you for your comments, and we agree that the new policy might not necessarily be a faithful explanation of the original policy. Constructing a faithful explanation is an interesting problem, but would also require a different algorithm. Instead, our goal is to train a new, interpretable policy that can be used as a substitute for the original blackbox policy, while still preserving performance. This underlies the reason why we aim to maximize rewards while having an interpretable architecture, rather than imitating the actions exactly.

How are Appendix L's maximum abstraction errors computed? Are they evaluated on held-out states, and do they quantify the error introduced by relaxing (2d)?

Thank you for the question on the abstraction errors from Appendix L. We agree that the theoretical guarantee applies to the original minimax formulation, and does not directly extend to the relaxed objective. Appendix L empirically quantifies the maximum abstraction error under this relaxation, evaluated on held-out states from separate rollouts. We will clarify this distinction and the evaluation procedure in the revised manuscript..

What concrete inspection or diagnosis task benefits from replacing the reference policy with the new concept-based policy? Please illustrate with a specific state, concept values, and action or correction.

As a concrete example of the benefits of concept-based models in practice, consider a medical setting where an automatic glucose agent recommends insulin dosing. Interpretability is critical here because the doctor should be able to see how different factors might impact the glucose agent. Pure blackbox models such as LLMs would be inappropriate here because it would be difficult to audit them, and doctors would have little understanding or trust that the algorithm would correctly perform its duties. Concept-based models in this setting could improve explainability and allow doctors to understand why actions were taken by the policy.

In our scenario, the state space would be the raw medical information consisting of glucose history, blood-related data, time-of-day, information on the meals eaten, etc., and the set of concepts would be binary signals such as whether glucose has risen, and whether we have eaten a meal recently. For example, if glucose is rising and a meal was recently eaten, the policy might recommend increasing insulin delivery. If the meal concept were incorrectly predicted, a doctor could correct that concept and observe whether the recommended action changes. The key benefit of a concept-based policy is that it allows for human understanding and potential correction or debugging when errors are made.

Reviewer_53k7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the problem of selecting a small subset of human-interpretable concepts for reinforcement learning without repeatedly training policies for candidate subsets. To address this issue, the paper presents DRS (decision-relevant selection), selecting concepts that distinguish states with different action values, and its variant DRS-log. The paper provides theoretical analysis that connects concept selection to state abstraction. Experiments evaluate DRS and its variant across five RL environments, and experimental results show performance gain.

Critical strengths
  • The paper investigates a meaningful problem in concept-based reinforcement learning.
  • The paper presents experimental results showing promising evidence that significant concept reduction is possible.
  • The paper provides theoretical analysis, showing the performance guarantee for perfect concept predictors.
Critical weaknesses
  • Although this paper is attempting to provide theoretical performance guarantee for both perfect and imperfect concept predictors, the paper shows that it cannot offer theoretical performance guarantee for imperfect concept predictors.
  • While experimental results in the paper indicate the improved performance, the performance gain is very marginal, leading to the questionable effectiveness.
  • The concept-based reinforcement learning is seemingly only suitable for small-scale systems, as it still requires manual efforts for constructing the concept bank. Thus, the practicability is questionable.
  • The paper mathematically formulates the DRS problem using Eqs. (2a) – (2f). However, it misses the description of the formulated problem, making it difficult to follow.
Questions for authors
  1. Is Y a variable that needs to be determined by solving the formulated problem? Please clarify it.
  2. What controls \epsilon’ after removing eq. (2d)?
  3. The description of DSR-log is not adequate. Please add more explanations.
  4. What are policy inputs? Are they raw logits, probabilities, or binary predictions in the experiments?
3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Dear Reviewer 53k7,

Thank you for your review, and for viewing our work as addressing a meaningful problem in concept-based reinforcement learning, with promising evidence that significant concept reduction is possible and theoretical analysis showing the performance guarantees. We provide answers to your questions below.

Our paper introduces the concept selection problem and designs algorithms to identify and select the most decision-relevant concepts. For concept-based models, identifying the set of concepts to use from a concept bank is a labor intensive process. To automate this, we introduce decision-relevancy as the main criteria for concept selection and show how to identify decision-relevant concepts in both perfect and imperfect settings.

Although this paper is attempting to provide theoretical performance guarantee for both perfect and imperfect concept predictors, the paper shows that it cannot offer theoretical performance guarantee for imperfect concept predictors. The description of DSR-log is not adequate. Please add more explanations.

In Section 4.3 we extend this setting to imperfect concept predictors. The key difference with imperfect concept predictors is that separation between state pairs, Ys,s′Y_{s,s'} is now probabilistic with probability bounded by the accuracy of the concept predictors. We bound this probability using a logarithmic transform, looking in particular at log⁡(1−Ys,s′)≥∑j:1[cj(s)≠cj(s′)]Kxjlog⁡(1−(δj2+(1−δj)2))\log\left(1-Y_{s,s'}\right) \ge \sum_{j : \mathbf{1}[c_j(s)\neq c_j(s')]}^{K} x_{j} \log\bigl(1-(\delta_j^2+(1-\delta_j)^2)\bigr). We note that in Lemma 4.1, we provide theory for the imperfect concept predictor setting, as we demonstrate how adversarial loss can lead to maximally poor performance.

We next describe a new extension that allows us to give performance bounds under the assumption that concept predictors make independent errors. Let c\mathbf{c} be the set of selected concepts, and let δj\delta_{j} be the accuracy of the concept predictor jj. Then running DRS under perfect concept predictors yields that Vπ∗(s)−Vπc(s)≤2ϵ(gc)(1−γ)2V^{\pi^*}(s)-V^{\pi_c}(s) \le \frac{2\epsilon(g_c)}{(1-\gamma)^2}.

Next, consider the policy π^c\hat{\pi}_c that takes actions with imperfect rather than perfect concept predictors, but otherwise uses the same mapping from concept to action. Assume that, conditional on any state, each concept predictor is correct with probability at least δj\delta_j and prediction errors are independent across concepts.

We first bound Vπc(s)−Vπ^cV^{\pi_c}(s) - V^{\hat{\pi}_c}. At any state s\mathbf{s}, the probability that π^c\hat{\pi}_c and πc\pi_c predict a different action is at most 1−∏jδj1-\prod_{j} \delta_j, because at least one concept predictor must make an error. The discounted reward is at most Rmax⁡1−γ\frac{R_{\max}}{1-\gamma}, and so by the performance difference lemma, we have that Vπc(s)−Vπ^c≤Rmax⁡(1−∏jδj)(1−γ)2V^{\pi_c}(s) - V^{\hat{\pi}_c} \leq \frac{R_{\max} (1-\prod_{j} \delta_j)}{(1-\gamma)^2}.

Combining gives that we can bound policy performance between the optimal policy and a policy trained on concepts selected through DRS as Vπ∗(s)−Vπ^c(s)≤2ϵ(gc)+Rmax⁡(1−∏jδj)(1−γ)2V^{\pi^*}(s)-V^{\hat{\pi}_c}(s) \le \frac{2\epsilon(g_c) + R_{\max} (1-\prod_{j} \delta_j)}{(1-\gamma)^2}. This relates the accuracy of individual concept predictors to the performance of policies trained on these concepts.

While experimental results in the paper indicate the improved performance, the performance gain is very marginal, leading to the questionable effectiveness.

Thank you for your questions on our experiments. Our key empirical results are that selecting concepts in a decision-relevant manner can improve the reward of the learned policy compared to baselines. Figure 3 demonstrates this across different environments, both in the perfect and imperfect scenario, and Figure 9 in Appendix F shows that this holds even when using a smaller number of concepts. For example, in Pong, DRS and DRS-log achieve within 11% of the optimal reward using only 8% of the concepts (19 out of 228), while all baselines achieve less than 50% of the optimal reward at the same number of concepts. We appreciate being able to highlight these results, and will bring the figure to the main body of the paper in a revised version.

AuthorsOpenReview ↗

Rebuttal Part 2

The concept-based reinforcement learning is seemingly only suitable for small-scale systems, as it still requires manual efforts for constructing the concept bank. Thus, the practicability is questionable.

We follow the assumption that is standard within the concept-based literature that a concept-bank is given [1,2,3]. We additionally note that the construction of the concept bank can be automatically done through LLMs [4]. To demonstrate this, we run a new experiment where we automatically generate concept banks using LLMs for the perfect MiniGrid environment. We post results below, and find similar results to the hand-crafted concept bank scenario. DRS achieves an optimal performance of 0.960.96, which is 29% better than the best baseline algorithm.

MethodConcepts usedReturn
All concepts500.9570 ± 0.0046
Random110.7465 ± 0.0435
Variance110.6848 ± 0.2108
Greedy110.4403 ± 0.0518
DRS110.9606 ± 0.0008

[1] Ye, et al., LICORICE: "Label-Efficient Concept-Based Interpretable Reinforcement Learning" (ICLR 2025) [2] Zabounidis, Renos, et al. "Concept learning for interpretable multi-agent reinforcement learning." (CoRL 2023) [3] Delfosse, Quentin, et al. "Interpretable concept bottlenecks to align reinforcement learning agents." (NeurIPS 2024) [4] Oikarinen et al., "Label-free concept bottleneck models" (ICLR 2023)

The paper mathematically formulates the DRS problem using Eqs. (2a) – (2f). However, it misses the description of the formulated problem, making it difficult to follow. What controls \epsilon’ after removing eq. (2d)?

We start from the perfect setting and develop the DRS algorithm. The key idea behind DRS is that minimizing the abstraction error ensures that the concepts we select are decision-relevant. Formally, equation 2a minimizes the state abstraction error, Ds,s′D_{s,s'} for pairs of states that are not separated by any concept, 1−Ys,s′1-Y_{s,s'}. Pairs of states with the same concept representation should have similar Q values.

Equation 2b then sets a cap on the maximum number of concepts selected. Equations 2c and 2d help define Ys,s′Y_{s,s'}. Equation 2c defines Ys,s′Y_{s,s'} based on whether any selected concept (xj=1x_j=1) has differing values for some concept (cj(s)≠cj(s′)c_{j}(s) \neq c_{j}(s')). Equation 2d enforces an ordering over separated state pairs: if Ds,s′>Dt,t′D_{s,s'} > D_{t,t'} and Yt,t′=1Y_{t,t'}=1, then Ys,s′=1Y_{s,s'}=1 as well. That is, separating a lower-error pair requires first separating all the higher error pairs.

Finally, Equation 2e enforces that we have different concept representations for ρ\rho-fraction of the state pairs with different actions; this way the learned policy can separate these states. We will include this expanded description in an updated version of the paper.

In practice, we relax equation 2d. Doing so avoids minimizing the maximum abstraction error between states with the same concept representation (Ds,s′(1−Ys,s′)D_{s,s'} (1-Y_{s,s'})), and instead minimizes the average abstraction error across those same states with the same concept representation. In essence, this trades off a worst-case approach for an average-case one, which leads to better performance in practice.

Is Y a variable that needs to be determined by solving the formulated problem? Please clarify it.

In our optimization problem in Equation 2a, Y is a decision variable that also needs to be solved along with x. It represents whether a pair of states, ss,s′s', are separated by any concept.

What are policy inputs? Are they raw logits, probabilities, or binary predictions in the experiments?

The inputs to the policy are raw logits from the concept predictors.

We hope we have answered all your questions. Please let us know if you have any further questions; we are more than happy to answer any additional questions you might have.

Reviewer_NKGrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes an optimization-based selection algorithm for human-interpretable concepts that can be used for training concept-based policies. Two versions of the optimization problem are proposed that distinguish between noisy and perfect concepts. The proposed selection mostly improves the performance of concept-based policies on standard RL benchmarks compared to other heuristics and methods of selecting concepts. Further, the proposed approach seems to be robust to imperfections in value functions, shows more improvements from manually improved concepts, and matches the selection performance of a human selector on the CUB benchmark.

Critical strengths
  • Interpretability of neural policies is a difficult problem, and extracting decision-relevant concepts is a valid solution
  • The noisy predictor version of the optimization problem is interesting and potentially allows to extend to broader settings
  • Comprehensive benchmarking of the proposed selection strategy
Critical weaknesses
  • The proposed solution likely won't scale to more complex tasks where the states need to be discretized finely (and with noisier Q-functions) as already the computation time is in the range of minutes
  • The necessity of the derivations is not fully clear, especially in Sec. 4.2
  • Assumptions should be stated more clearly, and their validity needs to be argued, e.g., access to optimal policy and Q function
Questions for authors
  • The introduction should be more concise in introducing "concept". Currently, different examples are used, which are often only partially explained. It is also unclear what of the Fig. 1 pipeline is the focus of the paper
  • How is the concept accuracy presented after proposition 3.1. connected to the previous framework? I don't see any introduction of δ\delta
  • Why is Thm 4.1. showing performance? It seems like a tautology as it assumes the optimal solution. In other words, what do I learn that I don't know without the Theorem?
  • For me, it is unclear if 4.2. adds much since the additional variables introduced will be hard to estimate, besides benchmarks, and the results seem straightforward from Theorem 2.1. Can you explain why this section carries important insights?
  • Why is it sufficient to sample from states for the optimization? Also, what are abstract states exactly?
  • The legend of Fig 4 is confusing. Is k=11 for the left plot?
  • Why do you think your method is performing worse for pong in the perfect setting?
  • The intervention testing was particularly difficult to follow. Can you clarify what happens and why it is ensured that a concept predictor improves?
  • It seems that your work is restricted to discrete actions to keep the Q evaluations over A feasible. Is this correct? Could there be extensions to continuous actions?
  • How realistic is the assumption of a concept bank, which are essentially atomic propositions? What is the relationship of this work to building decision trees from neural networks [1,2,3]?
  • Can you differentiate your work more clearly from existing concept-based RL research [4] and works cited in Sec. 6. Is the main difference using Q functions to decide, aka decision relevance?

[1] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. 2018. Verifiable reinforcement learning via policy extraction. https://dl.acm.org/doi/10.5555/3327144.3327175

[2] Pranav Ashok, Mathias Jackermeier, Jan Křetínský, Christoph Weinhuber, Maximilian Weininger, and Mayank Yadav. 2021. DtControl 2.0: Explainable Strategy Representation via Decision Tree Learning Steered by Experts. https://doi.org/10.1007/978-3-030-72013-1_17

[3] Daniël Vos and Sicco Verwer. 2023. Optimal decision tree policies for Markov decision processes. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI '23). Article 606, 5457–5465. https://doi.org/10.24963/ijcai.2023/606

[4] Quentin Delfosse, Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. 2023. Interpretable and explainable logical policies via neurally guided symbolic abstraction.

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Dear Reviewer NKGr,

Thank you for your response, and we appreciate that you find the interpretability of neural policies a difficult problem, that our solution to the noisy version of the problem interesting, and that we comprehensively benchmark our proposed selection strategy. We provide clarifications and answers to your questions below.

The proposed solution likely won't scale to more complex tasks where the states need to be discretized finely (and with noisier Q-functions) as already the computation time is in the range of minutes

We train policies on the concept space, which are less fine-grained than the raw state space but interpretable. We refer to this concept space as the “abstract state space”; we will update the manuscript to use concept space throughout to avoid confusion. We sample concept combinations from rollouts and optimize equation 2a using these samples, with the goal of estimating the distribution of differences in Q values between pairs of states By sampling rather than exhaustively searching, we can scale our method to larger state spaces, and our method is only performed once upfront.

The necessity of the derivations is not fully clear, especially in Sec. 4.2 Why is Thm 4.1. showing performance? It seems like a tautology as it assumes the optimal solution. In other words, what do I learn that I don't know without the Theorem? For me, it is unclear if 4.2. adds much since the additional variables introduced will be hard to estimate, besides benchmarks, and the results seem straightforward from Theorem 2.1. Can you explain why this section carries important insights?

We appreciate the chance to clarify Theorems 4.1 and 4.2. At a high level, Section 4.2 shows how we can operationalize decision-relevant selection as an optimization problem. Theorem 4.1 relates the optimization problem from 2a to the performance of a concept predictor using those concepts, and shows that we can reduce the problem of finding the optimal set of concepts to solving a linear program. We only assume that we can solve this 0-1 linear program optimally. The key insight is that we can reduce finding a decision-relevant set of concepts to solving a particular linear program. Theorem 4.2 demonstrates that using approximate Q values in equation 2a should suffice because errors in Q estimation propagate linearly to policy reward. We will update the manuscript to emphasize and clarify this.

Assumptions should be stated more clearly, and their validity needs to be argued, e.g., access to optimal policy and Q function

Thank you for the suggestion, and we will make our assumptions on the optimal policy and Q values more explicit throughout. In Appendix N, we show that sub-optimal policies trained for only 100k timesteps (instead of 500k) still lead to good performance for DRS. Theorem 4.2 establishes robustness to bounded Q-value estimation error, while Appendix N empirically demonstrates robustness to suboptimal reference policies.

The introduction should be more concise in introducing "concept". Currently, different examples are used, which are often only partially explained. It is also unclear what of the Fig. 1 pipeline is the focus of the paper

Thank you for the suggestion, and we will focus on a single example in more detail in the introduction. A concept is a human-interpretable property of an input; for example in an environment like Pong, it could be whether the coordinate of the paddle is greater than zero. Figure 1 illustrates the typical pipeline of using concepts, and we focus on the concept selection stage; we will make this clearer in the introduction.

How is the concept accuracy presented after proposition 3.1. connected to the previous framework? I don't see any introduction of

Let S\mathcal{S} be the state space, and Ti⊂ST_i \subset \mathcal{S}. We can define an imperfect concept predictor as fcTf_{\mathbf{c}}^{T}, so that fciTi(s)=1−ci(s)f^{T_i}_{c_i}(s) = 1 - c_i(s) for s∈Tis \in T_i; that is, the concept predictor errors for s∈Tis \in T_i. Then we can define δi\delta_i as the accuracy of this predictor; δi=1−∣Ti∣∣S∣\delta_{i} = 1-\frac{|T_i|}{|\mathcal{S}|}. The key intuition here is that a δi\delta_{i}-accurate concept predictor makes mistakes on 1−δi1-\delta_{i} fraction of the states.

The legend of Fig 4 is confusing. Is k=11 for the left plot?

Figure 4 studies how robust our method is to noisy concepts. On the left, we fix k=11, while varying the concept predictor accuracy between 75%-95%. On the right, we fix concept predictors to be 95% accurate, while varying kk from 3 to 11. Increasing concept accuracy improves training efficiency, while increasing the number of concepts increases the maximum performance.

AuthorsOpenReview ↗

Rebuttal Part 2

Why do you think your method is performing worse for pong in the perfect setting?

In Pong, fine-grained positional control is important for optimal decision-making, which is difficult to capture with binary concepts regardless of which subset is selected. As a result, all methods are constrained by the expressiveness of binary concepts in the perfect setting, making it difficult to achieve good performance. Notably, baselines do not perform much better than DRS either, suggesting this reflects a general limitation of binary concepts in Pong rather than a specific weakness of DRS.

The intervention testing was particularly difficult to follow. Can you clarify what happens and why it is ensured that a concept predictor improves?

For test-time intervention, we follow the standard paradigm from the concept-based model literature [1,2,3]. Test-time intervention refers to an oracle fixing mispredicted concepts. The goal is to show that, in theory, a person using a concept-based model could see performance improvements if they correct concept mispredictions. In Figure 6, we simulate test-time intervention by first randomly selecting 50% of concepts to intervene on, then correct concept predictions for these concepts in all states. We compute the resulting accuracy of the concept predictors and reward for the policies, and find that DRS achieves high reward both pre- and post-intervention.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." (ICML 2020)

[2] Espinosa Zarlenga, Mateo, et al. "Concept embedding models: Beyond the accuracy-explainability trade-off." (NeurIPS 2022)

[3] Espinosa Zarlenga, Mateo, et al. "Learning to receive help: Intervention-aware concept embedding models." (NeurIPS 2023)

It seems that your work is restricted to discrete actions to keep the Q evaluations over A feasible. Is this correct? Could there be extensions to continuous actions?

Throughout our work, we focus on discrete actions. Continuous actions are an interesting extension, though complicates the algorithm from Section 4.1, as we would need to optimize over a continuous rather than discrete space. However, if we could find an efficient optimizer to find the maximum discrepancy in Q values, given a state, then we could extend the DRS algorithm to this setting. We will include discussion on how our results can be extended to continuous actions in an updated version of the manuscript.

How realistic is the assumption of a concept bank, which are essentially atomic propositions?

We agree that the question of where the concept bank comes from is an important and distinct line of work. We follow the assumption that is standard within the concept-based literature that a concept-bank is given [4,5,6]. We run a new experiment where we automatically generate concept banks using LLMs for the perfect MiniGrid environment. We post results below, and find similar results to the hand-crafted concept bank scenario. DRS achieves an optimal performance of 0.960.96, which is 29% better than the best baseline algorithm.

MethodConcepts usedReturn
All concepts500.9570 ± 0.0046
Random110.7465 ± 0.0435
Variance110.6848 ± 0.2108
Greedy110.4403 ± 0.0518
DRS110.9606 ± 0.0008

[4] Ye, et al., LICORICE: "Label-Efficient Concept-Based Interpretable Reinforcement Learning" (ICLR 2025)

[5] Zabounidis, Renos, et al. "Concept learning for interpretable multi-agent reinforcement learning." (CoRL 2023)

[6] Delfosse, Quentin, et al. "Interpretable concept bottlenecks to align reinforcement learning agents." (NeurIPS 2024)

What is the relationship of this work to building decision trees from neural networks?

Our work focuses on the question of how we can select the most decision-relevant subset of concepts before we train concept-based policies. Concept-based RL requires a set of pre-defined concepts, but much of the other work in this space treats determining the set of concepts to use as a manual annotation problem, while our work seeks to automate this process. In our work, Q values are used as a means to achieve this end and to quantify decision-relevance. Our work is complementary to [7], as the set of selected concepts we identify could be used for policy training; we could have decision-tree policies as the policy in a concept-based model, but our focus is on how we select concepts prior to this.

[7] Delfosse, Quentin, et al. "Interpretable and explainable logical policies via neurally guided symbolic abstraction" (NeurIPS 2023)

Reviewer_NKGrOpenReview ↗

Response to rebuttal

Dear authors, Thanks for your response and the revision. I have a few follow-up questions and comments:

  • My question regarding related methods using decision trees instead of concepts was referring to the literature (i.e. Bastani et al and related works) that distills decision trees from policies. What contribution does your approach provide compared to these works?
  • I appreciate the evaluation with imperfect Q functions. However, the current analysis does not fully support the claim of robustness under uncertainty since the level of imperfection is unclear. An ablation study that adds noise (different types and magnitudes) to the Q function and evaluates degradation would be necessary to fully support the claims.
  • I still think that Sec. 4.2 and the response do not clearly state the gap usefulness of the derivations. Can you be more specific about how you will adapt this section, and make it clearer why the theoretical analysis is neither trivial nor impactless?
  • While I appreciate a discussion of future work with continuous action spaces. I find the added sentence in Sec. 3.2 unconvincing and would suggest removing it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.