acceptodds
Under review as a conference paper at ICLR 2027

Faithful Rules Are Not Enough: Boolean Surrogates for Deep RL and an Audit of Their Concepts

Abstract

Rule surrogates are among the most readable explanations of deep reinforcement learning (DRL) policies, and their quality is almost always judged by a single number: how often the rules reproduce the policy's actions. We ask whether that number says anything about the concepts the rules are written in. We first build a strong surrogate for vision-based policies. LUCID jointly trains a sparse autoencoder on the frozen policy's features, a learnable binarization bottleneck and a product t-norm logic layer, and extracts Disjunctive Normal Form (DNF) rules; once the selectors polarize, the extracted rules are invariant over a wide range of extraction thresholds, and a state-level certificate bounds the disagreement between the rules and the continuous surrogate. Across five vision-based tasks, from MiniGrid to Pong and Boxing, the rules agree with the PPO teacher on 96.8–100% of states and need about 2× fewer feature tests than a decision tree of matched fidelity on gridworlds and 28–46× fewer on Atari; controls show that joint training is what keeps them faithful after extraction. We then audit the concepts along two axes: alignment with simulator ground-truth factors, and causal effect under intervention on the frozen teacher. Under our criteria the two never hold together in any of the five environments, and on Dynamic-Obstacles rules with near-perfect fidelity rest on concepts whose manipulation moves the teacher away from the predicted action. VLM-generated concept labels, tested blind, predict their concepts above chance but well below certainty. Behavioural fidelity, factor alignment and causal efficacy are distinct properties, and rule-extraction work should report all three.

Primary area:
interpretability and explainable AI
Keywords:
explainable reinforcement learning, post-hoc surrogate, rule extraction, disjunctive normal form, sparse autoencoder, concept discovery, neuro-symbolic, faithfulness evaluation, causal intervention, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GZcFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies post-hoc rule extraction for vision-based deep RL policies. It proposes LUCID, a method that jointly trains (i) a sparse TopK autoencoder (SAE) on a frozen policy's CNN features, (ii) a learnable per-concept binarization bottleneck, and (iii) a product t-norm DNF logic layer, all optimized end-to-end to imitate the teacher's action distribution. Once the per-literal selector probabilities polarize during training, Boolean DNF rules can be extracted at essentially any threshold in a provably invariant interval (Proposition 1), and a state-level "soft–hard" certificate (Theorem 1) upper-bounds the disagreement between the continuous surrogate and the extracted discrete rules.

The empirical contribution has two parts:

  1. Fidelity/compactness: across five vision-based control tasks (two MiniGrid tasks, PixelCartPole, Atari Pong, Atari Boxing), LUCID's extracted rules agree with the PPO teacher on 96.8–100% of states while using far fewer feature tests (literals) than decision-tree baselines (VIPER, SA-DT, Soft-DT) at matched fidelity — 2× fewer on gridworlds and 28–46× fewer on Atari.
  2. A concept audit that separates "does a concept match a human-nameable simulator factor" (factor alignment, via AUC against ground-truth predicates) from "does a concept causally drive the decision" (causal efficacy, via concept-editing interventions on the frozen teacher). The main empirical finding is negative: these two properties never co-occur across all five environments, and on Dynamic-Obstacles a near-perfectly faithful rule set is empirically shown to be driven by concepts whose causal intervention actually moves the teacher away from the predicted action. A VLM-based concept-naming protocol (namer + independent scorer, blind evaluation) is introduced as a tertiary check, showing automatically generated labels beat distractors but remain well below certainty.
Critical strengths
  1. The approach of distinguishing between "does it look right?" and "is it actually correct?" is highly commendable. Many papers claim a strategy is "interpretable" simply because rule extraction yields a match rate exceeding 96%. However, the authors of this paper pause to ask two critical questions: Do the "concepts" within these rules actually correspond to human-understandable notions (such as "the key is in sight")? And do these concepts genuinely drive decision-making, or are they merely window dressing? They conducted specific causal intervention experiments to verify the latter point. They discovered that across five different environments, the conditions of "concepts making sense" and "concepts actually being effective" never held true simultaneously. In some environments, the rules appeared remarkably accurate, yet closer inspection revealed that swapping the active concept would cause the strategy to move in the opposite direction. This valuable finding serves as a wake-up call for the entire field of rule-based interpretability.

  2. The ablation study regarding the methodology itself is meticulously executed, convincingly demonstrating the genuine utility of the joint training approach. The authors compared a "train concepts first, then extract rules" (two-stage) method against a "train concepts and rules simultaneously" (joint) method. They found that while the two-stage method could achieve over 90% imitation accuracy, converting the extracted rules into rigid 0/1 logic caused fidelity to plummet by 14–47 percentage points, while also requiring thousands of decision conditions; in contrast, the joint training approach required only a few dozen. This demonstrates that the design choice of "joint training" was not merely a way to massage the data, but a genuine solution to the problem.

Critical weaknesses
  1. The core negative conclusion—that the "rules make sense but don't work"—was based on tests across only five games or environments. Furthermore, the authors themselves acknowledged the presence of confounding factors (such as variations in the granularity of "human conceptual vocabularies" and vast differences in the number of possible actions across environments). While the conclusion sounds striking, the limited sample size and confounding variables suggest the results might differ in other settings; thus, the finding should be viewed with skepticism rather than accepted as a universal rule.

  2. The high fidelity observed in one environment (DoorKey) is somewhat misleading. In that game, the strategy involved simply moving forward by default more than half the time; the rules didn't actually make a reasoned judgment but rather defaulted to the most common action—which happened to be correct. Although the abstract boasts a "96.8%–100% fidelity" rate, a significant portion of that high score was achieved simply by "guessing the most common answer," rather than the rules actually learning any underlying logic.

Questions for authors
  1. Regarding the conclusion that "a rule cannot be both intuitive and effective," could you test a few more environments—or at least control for confounding factors like rule granularity and the number of actions—before drawing a final conclusion? Right now, it feels a bit like a "hasty, drastic fix."

  2. For the DoorKey environment, could you report separate figures for successes achieved via "fallback actions" versus those achieved through "actual rule-based reasoning," rather than glossing over the distinction by lumping them into a single fidelity percentage?

  3. Regarding the control of variables: so far, you’ve only tested PPO as the teacher model. If you were to switch to a value-based strategy like DQN, would the argument for the stability of this joint training approach (specifically, how SAE sparsity prevents training collapse) still hold water?

4 comments

We thank the reviewer for the careful reading and the precise questions. We answer them in the order of the review. A revised PDF has been uploaded, with all new or changed text marked in blue.

W1 & Q1: The divergence between alignment and causal efficacy rests on five environments with confounds

We agree with the reviewer, and we have revised both the claim and the evidence.

(a) Narrower claim. The paper already presents the divergence as "an observation, not a law" (Section 5 and Appendix E.8). Some sentences still read more strongly than that, in particular the abstract ("never hold together") and the conclusion. We rewrote them to state what was tested: the abstract now says that under our criteria the two "did not hold together in any of the five environments we tested", and the introduction and conclusion state that they did not hold together in any of the ten teacher–environment pairs we tested, including the DQN pairs below. The revision also states that this is an empirical finding about LUCID's dictionaries, not a general property of rule surrogates.

(b) More cases. Following the reviewers' suggestions, we repeated both audits with DQN teachers on five environments: the four original environments where DQN solves the task, plus Enduro, a new and visually richer Atari game. This gives five new teacher–environment pairs. On the four original environments the factor vocabularies and action spaces are unchanged while the teacher, and with it the features the dictionary is learned from, changes. Within the rebuttal period the DQN audits use one LUCID seed (42), so the causal verdict for DQN is "CI > 0 on seed 42" rather than "in all three seeds"; we mark these verdicts as provisional and will report three seeds in the camera-ready.

EnvironmentTeacherAligned (G1 LUCID vs. CNN, cleaner 2nd-best AUC)Causal (clause necessity CI > 0; PPO: 3/3 seeds, DQN: seed 42)
Dynamic-Obs-5x5PPO / DQNyes / no (G1 44.4 vs. 11.1, but 2nd-best AUC not cleaner)no / no (−0.131)
PixelCartPolePPO / DQNno / no (0.7 vs. 1.7)yes / no (+0.003)
PongPPO / DQNno / no (3.2 vs. 5.3)yes / no (−0.018)
BoxingPPO / DQNno / no (0.6 vs. 2.4)yes / no (−0.013)
Enduro (new)DQNno (0.0 vs. 1.1)no (−0.013)

None of the new pairs is both aligned and causal, so the statement in its narrowed form holds for every pair we tested. We do not count the DQN pairs as strong new support for the divergence itself: with DQN on seed 42 neither property holds in any pair, so these pairs cannot show the two properties separating. What they do show is the paper's central point in its clearest form. LUCID's rules reach 96.7–99.7% hard fidelity on the DQN teachers (see Q3 and the response to DJUk), yet their concepts meet neither the alignment nor the causal criterion. Fidelity alone would have certified these surrogates; only the audits reveal the gap, which is why the paper recommends reporting fidelity, alignment and causal efficacy separately. The DQN pairs also show that, on the same environment with the same factors and action space, the verdicts change with the teacher: PixelCartPole, Pong and Boxing lose the causal verdict they have with PPO, and Dynamic-Obstacles loses its alignment verdict. We report these single-seed changes explicitly; they support reading the divergence as a property of particular teacher–dictionary pairs rather than of the environment alone, which is the narrower claim in (a).

W1 & Q1 (continued): the two confounds

(c) The two confounds.

  • Number of actions. Our causal effect is not a raw flip rate. It is the difference from a paired baseline that applies a perturbation of the same size to the same state, under the same teacher and the same action space (Section 5). Any mechanical advantage of having fewer actions is therefore shared by both arms of the comparison. What the action count can still change is the size of the difference, not its sign. This is why the paper classifies environments by whether the effect is significantly positive in every seed, not by its magnitude, and why we do not rank environments by effect size (Section 5). The sign is what separates the environments: on Dynamic-Obstacles, with 3 actions, the effect is significantly negative (sufficiency in 3/3 seeds, necessity in 2/3), while on Pong, with 6 actions, it is significantly positive in 3/3 seeds. The DQN results add a direct check: with the action space held fixed, the causal verdict on Pong, PixelCartPole and Boxing changes with the teacher, so the action count alone cannot explain it. Enduro adds a third action-space size (9 actions).
  • Granularity of the factor vocabulary. We agree that this confound cannot be removed with the current factors. Our factor vocabularies are coarse outside MiniGrid, and the teacher's own CNN units also align poorly with them on Atari. This suggests the vocabulary may be the binding constraint, but our data cannot separate that from a misaligned dictionary (Appendix E.6). We now name it explicitly in the main-text limitations as the most likely alternative explanation for the Atari results.

W2 & Q2: DoorKey fidelity, separating fallback from rule-based decisions

We thank the reviewer for this question. We agree that a single fidelity number hides the distinction, so we now report it split by who makes the decision (new Table 12, Appendix E.4). The table uses the hard policy on LUCID's validation split, with universal clauses removed: if no rule fires, the state goes to the fallback action (Forward, the teacher's most frequent action).

SeedStates decided by fallbackTeacher's Forward rate (majority baseline)Total fidelityFidelity points from rulesFidelity points from fallback
4253.04%52.96%99.76%46.8852.88
4353.27%53.15%99.88%46.7353.15
4454.07%53.99%99.92%45.9353.99

These numbers support three points.

  1. The rules account for the entire gain over guessing. A policy that always plays Forward reaches about 53%, while LUCID reaches 99.8–99.9%. All of the roughly 47-point difference comes from states where a rule fires.
  2. Where rules fire, they are almost always right. They agree with the teacher on 99.8%, 100% and 100% of the states they decide (seeds 42–44).
  3. The fallback is right because the rules are complete, not by luck. The fallback agrees with the teacher on 99.7–99.85% of the states it decides. That can only happen if the rules catch nearly every state where the teacher does not move forward, namely turning, picking up the key and opening the door. Rules plus fallback therefore form a decision list: "handle these exceptions, otherwise go forward". The default branch costs no literal, but it is not a guess. If a rule were missing, its states would fall to Forward and fidelity would drop.

We had reported this coverage effect in Table 2 (coverage 46.5% on DoorKey) and Section 5 (47.2% of decisions routed through a concept). We agree it was too easy to miss next to the headline range.

In the revision we make the following changes:

  • (i) We extend Table 2 in the main text with the majority-action baseline and the fidelity earned by firing rules, for all five environments (below). DoorKey is the only environment where the fallback decides a substantial share of states. In the other four it decides at most 5% of states, and almost all fidelity comes from firing rules.
EnvironmentStates decided by fallbackMajority-action baselineTotal fidelityFidelity points from rulesRule accuracy where rules fire
DoorKey-6x653.46 ± 0.54%53.37%99.85 ± 0.08%46.5199.94%
Dynamic-Obs-5x50.01 ± 0.02%43.24%99.99 ± 0.02%99.99100.00%
PixelCartPole4.95 ± 1.95%50.30%96.89 ± 0.96%94.7099.64%
Pong0.17 ± 0.14%23.53%99.78 ± 0.13%99.7799.95%
Boxing0.85 ± 0.45%26.19%98.40 ± 0.30%98.3399.18%

(Mean ± s.d. over three training seeds; same setting as the DoorKey table above. Total fidelity matches Table 2.)

  • (ii) In the abstract and Section 4, we state that on DoorKey about half of the states are decided by the default action.

Q3: Does the joint-training stability argument hold for a value-based teacher?

The argument concerns the surrogate, not the teacher. LUCID only ever sees the teacher's frozen features and its arg-max action. Whether those come from a policy head (PPO) or from the arg-max over Q-values (DQN), the training objective is a cross-entropy on hard action labels; for the DQN teachers we additionally weight each state by the teacher's Q-value margin, so that near-ties in Q do not dominate the loss. What can change is the geometry of the features, so the question is empirical. We answer it by repeating the controls of Table 4 on the DQN teachers:

  • two-stage: a reconstruction-only SAE, frozen before the DNF layer is trained;
  • no SAE: the DNF layer applied directly to the CNN features.

LUCID was re-tuned for each DQN teacher over a wide hyperparameter range, and each control was re-tuned separately over its own grid, which also includes LUCID's selected configuration. The controls use exactly LUCID's data, train/test split, Q-margin sample weights, clause-bias cap and selection rule.

Hard fidelity (%) on LUCID's held-out 10% of the teacher rollouts, seed 42; literals in parentheses. "LUCID's configuration" is the control trained with LUCID's selected hyperparameters.

Environment (DQN teacher)LUCIDTwo-stage, re-tunedNo SAE, re-tunedTwo-stage, LUCID's configurationNo SAE, LUCID's configuration
Dynamic-Obs-5x599.72 (10)98.06 (70)92.82 (50)88.32 (59)84.13 (72)
PixelCartPole96.88 (2)86.54 (25)85.27 (2,455)51.15 (2)71.37 (2)
Pong98.93 (52)77.37 (11,065)72.68 (11,386)51.48 (226)60.82 (243)
Boxing99.43 (178)73.93 (4,311)67.65 (61,237)65.83 (948)63.88 (783)
Enduro96.67 (255)59.89 (4,518)56.76 (54,843)54.14 (1,752)49.38 (1,650)

The result carries over from PPO. No re-tuned control reaches LUCID's fidelity on any DQN teacher, and each needs more literals to get as close as it does. The gap is small on Dynamic-Obstacles (98.1% vs. 99.7%, with 7× more literals), 10 points on PixelCartPole, where at LUCID's own size of 2 literals the controls reach only 51–71%, 22 points on Pong, where the best control uses over 200× more literals, 25 points on Boxing (24× more literals) and 37 points on Enduro (18× more literals). Dynamic-Obstacles is the one case where a control comes close, and we say so in the paper.

One feature of the DQN teachers is worth noting. Their CNN features are sparser than those of the PPO teachers. Counting units that are constant over the training rollouts, 336 of the 512 CNN units are inactive for the DQN teacher on PixelCartPole, against 216 for PPO, and 300 against 152 on Boxing. LUCID therefore learns its dictionary from fewer live dimensions, which makes these settings a useful stress test for the sparsity argument.

On sparsity specifically. The two controls bracket the question from both sides. The no-SAE control removes the sparse dictionary altogether and applies the DNF layer to the dense CNN features; even after re-tuning it stays 7–40 points below LUCID's hard fidelity on the DQN teachers (5–28 points on the PPO teachers). The two-stage control keeps a sparse SAE but trains it for reconstruction only, and it also falls short of LUCID on every DQN teacher. A sparse dictionary is therefore needed, but sparsity alone does not keep the rules faithful after extraction; what does is training the dictionary jointly with the rule layer. We do not ablate the TopK operator separately (for example, with a dense autoencoder trained jointly), so we cannot say how much of the effect is due to sparsity per se. We state this distinction in the revision.

A correction to the numbers cited in the review. The reviewer's second strength cites the gap between the controls and LUCID in Table 4. Following reviewer sMWe (Q4), we re-tuned both controls per environment instead of reusing LUCID's hyperparameters, and re-tuning narrows that gap. Over three PPO seeds, the best re-tuned control now stays 5–28 points below LUCID's hard fidelity (92.1% vs. 99.9% on DoorKey, 89.1% vs. 99.99% on Dynamic-Obstacles, 91.6% vs. 96.9% on PixelCartPole, 72.2% vs. 99.8% on Pong, 72.8% vs. 98.4% on Boxing), and on Atari it still needs about 30–260× more literals. The conclusion that joint training is what keeps the rules faithful stands, but with a smaller margin than in the submission, and Table 4 now reports the re-tuned numbers.

Reviewer_DJUkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method LUCID, authors train an end-to-end method to extract boolean rules for vision based RL. These rules are then evaluated on 3 criteria: behavioral fidelity, factor alignment, and causal efficacy. The broader question the paper answers is that reproducing actions alone dosent mean explaining its decisions, therefore high behavioral fidelity alone is not enough. The paper evaluates 5 task and report 96.8–100% fidelity and show factor alignment and causal efficacy vary depending on the environments.

Critical strengths
  1. The end-to-end training method is novel and interesting. Ablation also shows why end to end training is necessary. The VLM experiment separates namer from scorer, and through experiments show why a positive control is needed specifically for open weight vlms.

  2. The overall idea of evaluation is good. By measuring factor alignment and the causal effectiveness of concepts, the framework evaluates beyond the standard practice of just using fidelity. This evaluation strategy can be helpful for methods beyond stated in the paper.

Critical weaknesses
  1. The paper evaluated 5 task all with PPO teacher, the paper mentions this clearly but evaluating on one type of teacher doesn't generalise the finding of the paper to broader class of vision based RL. The results that we might be seeing may originate from using a specific type of teacher, thus I recommend authors to test their hypothesis to at least one other teacher method.

  2. The writing style of the whole paper could be improved, it is tough to read the paper at certain places. Also writing a one paragraph related work in the main paper and writing everything else in the ablation is not a good practice and limits the readability of the paper for broader audience. Therefore I would recommend authors to expand related work in main paper. Also the figure 1 is inconsistent with the text, embeddings are referred to as “x” in text and “e” in figure and the undefined term alpha appears in Fig 1 (Sparse encoder)

Questions for authors

Overall I am optimistic about the work. The authors should shed light on the weakness that I have mentioned.

4 comments

We thank the reviewer for the positive assessment and for the concrete suggestions. We address both weaknesses below. A revised PDF has been uploaded, with all new or changed text marked in blue.

W1: Results with a second type of teacher

We agree that a single teacher family limits the generality of the findings. We therefore added a value-based teacher, DQN, alongside the policy-gradient PPO teacher. The two cover the main families of model-free RL with discrete actions. The DQN teachers use the original algorithm as implemented in Stable-Baselines3, without Double, Dueling or prioritized replay. The Atari teachers use the RL Zoo hyperparameters. Each teacher is evaluated exactly like the PPO teachers in Table 7: deterministic actions, evaluation seeds 42–46, and mean ± s.d. over the five seed means.

EnvironmentDQN teacher returnPPO teacher return (Table 7)
Dynamic-Obs-5x5 (5 × 200 episodes)0.898 ± 0.0020.891
PixelCartPole (5 × 40 episodes)497.7 ± 2.8500.0
Pong (5 × 20 episodes)21.0 ± 0.021.0
Boxing (5 × 20 episodes)96.4 ± 0.792.2
Enduro, new (5 × 20 episodes)912.5 ± 54.6—

We report DQN on five environments: four of the original five, plus Enduro, a visually richer Atari game we added for reviewer sMWe. On DoorKey, vanilla DQN did not solve the task within our budget: the reward is sparse and the Q-values collapsed to zero. A teacher that does not solve the task has no meaningful behaviour to explain, so we keep only the PPO teacher there and state this in the paper.

We then ran the full pipeline on the DQN teachers. Because the teacher's features change, we re-tuned LUCID for each DQN teacher over a wide hyperparameter range. Three settings differ from the PPO runs, all following from the teacher rather than the surrogate, and all are reported in the new Appendix G: the clause-bias cap is matched to the extraction threshold, feature standardization is made robust to the many dead CNN units of these teachers, and the imitation loss weights each state by the teacher's Q-value margin, so that states where the teacher's choice is clear count more. On Boxing and Enduro LUCID is also trained on additional teacher rollouts. Each control was re-tuned separately over its own grid, which also includes LUCID's selected configuration, with the same data, sample weights, clause-bias cap and selection rule (best validation hard fidelity) as LUCID. Within the rebuttal period we trained LUCID with one seed (42) per DQN teacher; the camera-ready will report three seeds.

Results with the DQN teachers (seed 42; fidelity on LUCID's held-out 10% of the teacher rollouts; audit criteria as in Table 5):

EnvironmentSoft fidelity (%)Hard fidelity (%)LiteralsFactor alignment (G1, LUCID vs. CNN; aligned?)Causal necessity (clause level; CI > 0?)
Dynamic-Obs-5x599.5699.721044.4 vs. 11.1; no (2nd-best AUC not cleaner, p = 0.61)−0.131; no
PixelCartPole99.8496.8820.7 vs. 1.7; no+0.003; no
Pong99.0298.93523.2 vs. 5.3; no−0.018; no
Boxing99.3999.431780.6 vs. 2.4; no−0.013; no
Enduro98.5396.672550.0 vs. 1.1; no−0.013; no

Fidelity carries over. With DQN teachers LUCID's hard rules agree with the teacher on 96.7–99.7% of held-out states, the same range as with PPO (Table 4), with 2–255 literals. Appendix C.1 already stated that the method is agnostic to the RL algorithm and needs only the teacher's features and actions; these results are direct evidence for that claim.

The audits show the paper's central point most directly. With DQN (one seed), no pair passes either audit. On alignment, LUCID's dictionary has fewer monosemantic dimensions than the teacher's CNN units in four of the five environments, and on Dynamic-Obstacles, where it has more, its second-best factor AUC is not cleaner; on causal efficacy, switching off the clauses that fire does not reliably change the teacher's decision. Fidelity alone would have certified these rule sets as faithful; only the two audits show that their concepts carry neither the factor meaning nor the causal role one would read into them. This is exactly the case the paper's conclusion warns about (a rule set with near-perfect fidelity can rest on concepts that do not drive the teacher), and it is why we recommend reporting fidelity, alignment and causal efficacy separately. As with PPO, no pair is both aligned and causal. On the same environment the verdicts also change with the teacher, which supports stating the divergence only for the pairs we tested (GZcF W1). These are single-seed verdicts and are marked as such in the paper.

The new results are in the new Appendix G (Tables 19–25), and Section 4 summarizes them.

W2: Readability, related work and Figure 1

(a) Readability and related work.

We agree. We made two changes.

  • Related work in the main text. The single paragraph is now a full section with four paragraphs:

    • interpretable and neuro-symbolic RL;
    • post-hoc surrogates and concept bottlenecks;
    • discovering and naming concepts;
    • evaluating explanations beyond fidelity, which positions our audit against prior work on sanity checks and faithfulness.

    We added seven references, and the extended discussion stays in Appendix B.

  • Rewritten passages. We rewrote the passages that reviewers found hardest to read, without changing any equation, number or claim:

    • Section 3 is now split into Surrogate, Training objective and Why joint training is stable, and Eqs. 2–4 are explained in words.
    • The results of the concept audit are now split into Alignment and Causal efficacy.

If there are other passages the reviewer found difficult, we would be grateful for pointers and will revise them as well.

(b) Figure 1 inconsistent with the text.

Thank you for catching this. We redrew Figure 1 so that every symbol matches Section 3 and Appendix C:

  • Embedding notation: the observation is now oo and the embedding is xx throughout, instead of ee.
  • Undefined α\alpha: the SAE loss now reads λrec1d∥x−x^∥22+λ11D∥h∥1\lambda_{\mathrm{rec}}\tfrac{1}{d}\lVert x-\hat x\rVert_2^2+\lambda_1\tfrac{1}{D}\lVert h\rVert_1, as in Eq. 3.
  • Other symbols:
    • The bottleneck shows cj=σ(αj(zj−βj))c_j=\sigma(\alpha_j(z_j-\beta_j)) with αj=softplus(ρj)+12\alpha_j=\mathrm{softplus}(\rho_j)+\tfrac12, and cjhard=1[cj>θ]c_j^{\mathrm{hard}}=\mathbf 1[c_j>\theta]. The figure no longer shows a normalization of zz, which the method does not use.
    • The DNF layer uses the clipped bias b~a,i\tilde b_{a,i}.
    • The imitation loss is CE(softmax(s),abase)\mathrm{CE}(\mathrm{softmax}(s),a^{\mathrm{base}}).
    • The ramp ωt\omega_t appears in the total objective.

I appreciate the author response. I have gone through the new additions in the paper and it seems fine. I would like to maintain my score. And I lean towards acceptance.

Thank you for reading the revision and for your support. We appreciate your suggestions, which led to the DQN experiments and a clearer presentation of the paper.

Reviewer_sMWeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a post-hoc surrogate called LUCID for frozen vision-based reinforcement learning policies. It shows that, once the selectors have polarized, the extracted rule set remains unchanged over a central range of the selector threshold, and a state-level certificate bounds the disagreement between the continuous surrogate and the extracted rules. Tested in five environments, the rules reproduce the teacher's actions with high fidelity.

Critical strengths

The motivation of this work to tackle polysemantic convolutional features in vision-based policies.

The experimental design looks sound in terms of the four controls and comparing methods.

Critical weaknesses
  1. This work is tailored to discrete action spaces, which causes concerns of the overall contribution unfortunately. As acknowledged in the limitation discussion as well, discretizing multidimensional continuous actions leads to prohibitive growth, restricting its applications to important visual-control applications, such as robotics. This is an architectural limitation and narrows the methodological contributions and values, leaving broad claims about applicability to vision-based RL insufficiently supported.

  2. The scope of this work on vision-based policies is very focused, but the choice of evaluation environments is very restricted, offering limited evidence that proposed extraction and audit framework remains effective under richer visual variation, more complex representations, and larger concept vocabularies. Evaluation from more visually diverse settings is needed to support the claims about vision-based policies.

Questions for authors
  1. What is the fidelity of a policy that always takes the teacher’s most frequent action?

  2. Could you fit CART and VIPER directly on LUCID's binarized concepts chardc^{hard}, and provide the fidelity and size of these models? What role does the DNF layer play if the results from CART and VIPER match LUCID’s fidelity with a similar size?

  3. What is the model size of each of the five methods listed in Table 1 when using the internal node convention? If LUCID has larger size than VIPER in four of five environments, is the statement that LUCID is more compact than tree surrogate still accurate?

  4. The comparisons of the surrogates are inconsistent and incomplete. In Table 1 and Table 4, the tree sizes are measured using different approaches. The claim that joint training maintains extraction fidelity is not demonstrated on equal terms: the two-stage and no-SAE controls simply use LUCID's per-environment hyper-parameters, and no re-tuning is reported. The ablation studies do not isolate the DNF layer itself. A tree fitted on LUCID's own binarized concepts would be the natural control.

7 comments

We thank the reviewer for the detailed reading and for the four questions, which identified a real inconsistency in how we measured surrogate size. We have changed the paper in response: the size convention is now stated explicitly and used uniformly, the scope of the claims is narrowed, and we added the control the reviewer proposed (a tree fitted on LUCID's own binarized concepts). A revised PDF has been uploaded, with all new or changed text marked in blue.

W1: Restriction to discrete action spaces

We agree that the paper should state its scope precisely, and we have narrowed every claim to discrete-action, vision-based policies (abstract, introduction, conclusion and limitations). We would, however, like to separate two things that the weakness as stated runs together: the scope of the surrogate and the scope of the paper's contribution.

The restriction is a property of rule-based explanation, not of our architecture. A DNF rule set assigns clauses to actions: the statement "these concepts hold, therefore action aa" presupposes that aa is one of finitely many alternatives. Every rule or tree surrogate shares this restriction, which is why, to our knowledge, no rule-extraction method handles continuous actions from pixels. The closest work we are aware of, ORCAID, extracts DNF rules with affine action consequents for continuous control, but from low-dimensional state vectors (at most 17 dimensions). LUCID extends this literature along the input axis, from hand-specified state variables to learned concepts over pixels, rather than along the action axis. Within the rule-extraction literature its scope is therefore not narrower than the state of the art.

Discrete-action vision-based control is a substantial class. It covers the Atari suite, gridworld navigation, and the high-level discrete action layer of hierarchical robot policies. Our five environments span 2 to 18 actions.

The paper's central contribution does not depend on the action space. The main finding is negative and methodological: behavioural fidelity of 96.8 to 100 percent constrains neither whether the concepts name a simulator factor nor whether they causally drive the teacher, and the two properties did not hold together in any pair we tested. That finding is a statement about how concept-based surrogates should be evaluated. It applies to any surrogate whose atoms are learned concepts, including continuous-action ones, and including surrogates that are not DNF. The three evaluations we recommend reporting (fidelity on held-out states with online return, alignment against random-projection and PCA baselines, causal effect against a norm-matched perturbation) are defined independently of the action space.

We state the extension to continuous actions as the main open problem: it requires both a rule form with continuous consequents and a treatment of observations that mix pixels with proprioceptive vectors, which is the setting of most robot learning. We have rewritten the limitations section accordingly, and we no longer make unqualified claims about "vision-based RL".

W2: Restricted choice of evaluation environments

We added Enduro, a visually far richer Atari game, with the full pipeline: teacher, LUCID, both controls and both concept audits. Enduro changes its appearance during an episode, cycling through daylight, dusk, night and fog, with a variable number of cars on screen, so the same semantic situation appears under very different pixel statistics. It also has 9 actions, a third action-space size alongside our 2, 3, 6 and 18.

Enduro
Teacher (DQN, RL Zoo checkpoint)912.5 ± 54.6
LUCID soft fidelity98.53%
LUCID hard fidelity96.67%
Concepts / clauses / literals73 / 118 / 255
Decisions made by a firing rule100%
Best re-tuned control (two-stage)59.89% hard fidelity with 4,518 literals
Best tree baseline (return)Soft-DT, 720.4 ± 65.8 with 1,386 literals
LUCID hard return979.9 ± 72.2

Enduro is also the hardest case for the dictionary: 226 of the teacher's 512 CNN units are constant over the training rollouts, so LUCID learns its concepts from fewer than 300 live dimensions, and the extracted rules still reproduce the teacher on 96.7 percent of held-out states while every decision is routed through a concept literal rather than the default action.

We added one environment rather than several because the pipeline needs a teacher that actually solves its task: a teacher that does not has no coherent behaviour to explain, and auditing the concepts of a surrogate for an incompetent policy would tell us nothing. Enduro met that bar with a released checkpoint. Extending the evaluation to Atari games that require long-horizon exploration would first require training competent teachers for them, which was beyond what we could do within the rebuttal period, and we state it as the next step rather than claiming the current five environments settle the question.

On concept vocabulary size, the dictionary is already 512 concepts on PixelCartPole and the three Atari games (200 on Dynamic-Obstacles), of which 84 to 92 percent align with no factor in our vocabulary. We agree with the reviewer that a richer factor vocabulary for Atari is the main missing ingredient for the alignment audit, and we now name this explicitly in the main-text limitations as the most likely alternative explanation for the low Atari alignment, rather than as evidence that the dictionary is uniquely misaligned.

Q1: Fidelity of always taking the teacher's most frequent action

EnvironmentMajority actionMajority-action fidelityLUCID hard fidelity
DoorKey-6x6Forward53.4%99.9%
Dynamic-Obs-5x5Forward43.2%100.0%
PixelCartPoleRight50.3%96.9%
PongLEFT23.5%99.8%
BoxingLEFT26.2%98.4%

This column is now part of Table 2, together with the fidelity earned on states where a rule actually fires. In every environment the rules account for the entire gain over the majority baseline. DoorKey is the only environment where the default action decides a large share of states (53.5%), and there it is correct on 99.8% of them precisely because the rules capture nearly every state in which the teacher does not move forward; in the other four environments the default decides at most 5% of states. Reviewer GZcF asked the same question (W2/Q2) and Table 12 (Appendix E.4) gives the per-seed breakdown for DoorKey.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals

We are grateful to the three reviewers for their careful reviews. Their questions led to new experiments and to sharper claims. We summarize the outcome below.

Our paper strengths. LUCID jointly trains a sparse autoencoder, a binarization bottleneck and a DNF layer on a frozen vision-based policy, and then audits the concepts of the extracted rules along two axes that are usually conflated: alignment with simulator factors and causal efficacy on the teacher. The reviewers found the end-to-end training novel and interesting (DJUk), the controlled ablations meticulously executed and showing that joint training is a genuine solution (GZcF, DJUk), the motivation of tackling polysemantic CNN features well placed and the experimental design with its four controls sound (sMWe). They described separating "does it look right?" from "is it actually correct?" as a wake-up call for rule-based interpretability (GZcF), and an evaluation strategy that can help methods beyond the paper (DJUk).

Our main revisions and additional experiments.

  • Added a second teacher family: value-based DQN teachers on five environments, including Enduro, a new and visually richer Atari game, with the full pipeline: LUCID, re-tuned controls, tree baselines and both audits (Appendix G, Tables 19–25).
  • Re-tuned the two-stage and no-SAE controls per environment over their own grid, with three seeds (Table 4).
  • Fitted CART and VIPER on LUCID's own binarized concepts, which isolates the rule learner (Table 4, Tables 10–11).
  • Reported the majority-action baseline and the fidelity earned by firing rules versus the fallback action (Table 2; per seed for DoorKey in Table 12).
  • Unified the size convention (concepts and literals for every method, Table 9) and corrected the compactness claim accordingly.
  • Narrowed the claims to the teacher–environment pairs we tested and to discrete-action policies; expanded Related Work into a full section; rewrote Section 3 and redrew Figure 1 to match the notation.

We also justified some common questions as follows:

Do the findings depend on the type of teacher? (DJUk, GZcF)

  • Fidelity carries over. With DQN teachers, LUCID's rules agree with the teacher on 96.7–99.7% of held-out states with 2–255 literals, the same range as with PPO.
  • Joint training still matters. No re-tuned control reaches LUCID's fidelity on any DQN teacher: the best control is 1.7–36.8 points lower with 7–213× more literals.
  • The audits show the central point most directly. With DQN (one seed), no pair passes either audit, although the rules are highly faithful. Fidelity alone would have certified these rule sets; only the audits reveal the gap. In none of the ten teacher–environment pairs we tested do alignment and causal efficacy hold together, and on the same environment the verdicts change with the teacher, so we state the divergence for the pairs we tested rather than as a law.

How much of the fidelity comes from the rules rather than from a default action? (sMWe, GZcF)

  • Always predicting the teacher's most frequent action gives 23.5–53.4% fidelity, while LUCID reaches 96.9–99.99%; in every environment the firing rules account for the entire gain.
  • DoorKey is the only environment where the fallback decides a large share of states (53.5%). There it is correct on 99.7–99.85% of those states because the rules capture nearly every state in which the teacher does not move forward. The abstract now states this caveat.

Beside common questions, we address other questions in individual replies:

  • Re-tuned controls (sMWe): re-tuning raises the controls by up to 33 points, yet the best control stays 5–28 points below LUCID over three seeds and needs about 30–260× more literals on Atari.
  • Trees on LUCID's concepts (sMWe): most of the Atari advantage comes from the jointly learned concept space, while the DNF layer keeps the rules shorter on complex policies (Pong 19 vs. 106 literals, Boxing 93 vs. 227 against VIPER on the same concepts).
  • Discrete actions (sMWe): the restriction is shared by all rule surrogates; the evaluation methodology we recommend does not depend on the action space.
  • Visual diversity (sMWe): on Enduro the rules reach 96.67% hard fidelity with 255 literals, and every decision is made by a firing rule.
  • Confounds of the divergence (GZcF): the causal effect is measured against a paired baseline in the same action space; the coarseness of the factor vocabulary is now named in the limitations.
  • Stability with DQN (GZcF): a sparse dictionary is necessary but not sufficient; joint training with the rule layer keeps extraction faithful.
  • Readability (DJUk): Related Work is now a full main-text section, and Figure 1 matches Section 3.

We are happy to clarify any remaining point before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.