- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies post-hoc rule extraction for vision-based deep RL policies. It proposes LUCID, a method that jointly trains (i) a sparse TopK autoencoder (SAE) on a frozen policy's CNN features, (ii) a learnable per-concept binarization bottleneck, and (iii) a product t-norm DNF logic layer, all optimized end-to-end to imitate the teacher's action distribution. Once the per-literal selector probabilities polarize during training, Boolean DNF rules can be extracted at essentially any threshold in a provably invariant interval (Proposition 1), and a state-level "soft–hard" certificate (Theorem 1) upper-bounds the disagreement between the continuous surrogate and the extracted discrete rules.
The empirical contribution has two parts:
- Fidelity/compactness: across five vision-based control tasks (two MiniGrid tasks, PixelCartPole, Atari Pong, Atari Boxing), LUCID's extracted rules agree with the PPO teacher on 96.8–100% of states while using far fewer feature tests (literals) than decision-tree baselines (VIPER, SA-DT, Soft-DT) at matched fidelity — 2× fewer on gridworlds and 28–46× fewer on Atari.
- A concept audit that separates "does a concept match a human-nameable simulator factor" (factor alignment, via AUC against ground-truth predicates) from "does a concept causally drive the decision" (causal efficacy, via concept-editing interventions on the frozen teacher). The main empirical finding is negative: these two properties never co-occur across all five environments, and on Dynamic-Obstacles a near-perfectly faithful rule set is empirically shown to be driven by concepts whose causal intervention actually moves the teacher away from the predicted action. A VLM-based concept-naming protocol (namer + independent scorer, blind evaluation) is introduced as a tertiary check, showing automatically generated labels beat distractors but remain well below certainty.
Critical strengths
-
The approach of distinguishing between "does it look right?" and "is it actually correct?" is highly commendable. Many papers claim a strategy is "interpretable" simply because rule extraction yields a match rate exceeding 96%. However, the authors of this paper pause to ask two critical questions: Do the "concepts" within these rules actually correspond to human-understandable notions (such as "the key is in sight")? And do these concepts genuinely drive decision-making, or are they merely window dressing? They conducted specific causal intervention experiments to verify the latter point. They discovered that across five different environments, the conditions of "concepts making sense" and "concepts actually being effective" never held true simultaneously. In some environments, the rules appeared remarkably accurate, yet closer inspection revealed that swapping the active concept would cause the strategy to move in the opposite direction. This valuable finding serves as a wake-up call for the entire field of rule-based interpretability.
-
The ablation study regarding the methodology itself is meticulously executed, convincingly demonstrating the genuine utility of the joint training approach. The authors compared a "train concepts first, then extract rules" (two-stage) method against a "train concepts and rules simultaneously" (joint) method. They found that while the two-stage method could achieve over 90% imitation accuracy, converting the extracted rules into rigid 0/1 logic caused fidelity to plummet by 14–47 percentage points, while also requiring thousands of decision conditions; in contrast, the joint training approach required only a few dozen. This demonstrates that the design choice of "joint training" was not merely a way to massage the data, but a genuine solution to the problem.
Critical weaknesses
-
The core negative conclusion—that the "rules make sense but don't work"—was based on tests across only five games or environments. Furthermore, the authors themselves acknowledged the presence of confounding factors (such as variations in the granularity of "human conceptual vocabularies" and vast differences in the number of possible actions across environments). While the conclusion sounds striking, the limited sample size and confounding variables suggest the results might differ in other settings; thus, the finding should be viewed with skepticism rather than accepted as a universal rule.
-
The high fidelity observed in one environment (DoorKey) is somewhat misleading. In that game, the strategy involved simply moving forward by default more than half the time; the rules didn't actually make a reasoned judgment but rather defaulted to the most common action—which happened to be correct. Although the abstract boasts a "96.8%–100% fidelity" rate, a significant portion of that high score was achieved simply by "guessing the most common answer," rather than the rules actually learning any underlying logic.
Questions for authors
-
Regarding the conclusion that "a rule cannot be both intuitive and effective," could you test a few more environments—or at least control for confounding factors like rule granularity and the number of actions—before drawing a final conclusion? Right now, it feels a bit like a "hasty, drastic fix."
-
For the DoorKey environment, could you report separate figures for successes achieved via "fallback actions" versus those achieved through "actual rule-based reasoning," rather than glossing over the distinction by lumping them into a single fidelity percentage?
-
Regarding the control of variables: so far, you’ve only tested PPO as the teacher model. If you were to switch to a value-based strategy like DQN, would the argument for the stability of this joint training approach (specifically, how SAE sparsity prevents training collapse) still hold water?
4 comments
W1 & Q1 (continued): the two confounds
(c) The two confounds.
- Number of actions. Our causal effect is not a raw flip rate. It is the difference from a paired baseline that applies a perturbation of the same size to the same state, under the same teacher and the same action space (Section 5). Any mechanical advantage of having fewer actions is therefore shared by both arms of the comparison. What the action count can still change is the size of the difference, not its sign. This is why the paper classifies environments by whether the effect is significantly positive in every seed, not by its magnitude, and why we do not rank environments by effect size (Section 5). The sign is what separates the environments: on Dynamic-Obstacles, with 3 actions, the effect is significantly negative (sufficiency in 3/3 seeds, necessity in 2/3), while on Pong, with 6 actions, it is significantly positive in 3/3 seeds. The DQN results add a direct check: with the action space held fixed, the causal verdict on Pong, PixelCartPole and Boxing changes with the teacher, so the action count alone cannot explain it. Enduro adds a third action-space size (9 actions).
- Granularity of the factor vocabulary. We agree that this confound cannot be removed with the current factors. Our factor vocabularies are coarse outside MiniGrid, and the teacher's own CNN units also align poorly with them on Atari. This suggests the vocabulary may be the binding constraint, but our data cannot separate that from a misaligned dictionary (Appendix E.6). We now name it explicitly in the main-text limitations as the most likely alternative explanation for the Atari results.
W2 & Q2: DoorKey fidelity, separating fallback from rule-based decisions
We thank the reviewer for this question. We agree that a single fidelity number hides the distinction, so we now report it split by who makes the decision (new Table 12, Appendix E.4). The table uses the hard policy on LUCID's validation split, with universal clauses removed: if no rule fires, the state goes to the fallback action (Forward, the teacher's most frequent action).
| Seed | States decided by fallback | Teacher's Forward rate (majority baseline) | Total fidelity | Fidelity points from rules | Fidelity points from fallback |
|---|---|---|---|---|---|
| 42 | 53.04% | 52.96% | 99.76% | 46.88 | 52.88 |
| 43 | 53.27% | 53.15% | 99.88% | 46.73 | 53.15 |
| 44 | 54.07% | 53.99% | 99.92% | 45.93 | 53.99 |
These numbers support three points.
- The rules account for the entire gain over guessing. A policy that always plays
Forwardreaches about 53%, while LUCID reaches 99.8–99.9%. All of the roughly 47-point difference comes from states where a rule fires. - Where rules fire, they are almost always right. They agree with the teacher on 99.8%, 100% and 100% of the states they decide (seeds 42–44).
- The fallback is right because the rules are complete, not by luck. The fallback agrees with the teacher on 99.7–99.85% of the states it decides. That can only happen if the rules catch nearly every state where the teacher does not move forward, namely turning, picking up the key and opening the door. Rules plus fallback therefore form a decision list: "handle these exceptions, otherwise go forward". The default branch costs no literal, but it is not a guess. If a rule were missing, its states would fall to
Forwardand fidelity would drop.
We had reported this coverage effect in Table 2 (coverage 46.5% on DoorKey) and Section 5 (47.2% of decisions routed through a concept). We agree it was too easy to miss next to the headline range.
In the revision we make the following changes:
- (i) We extend Table 2 in the main text with the majority-action baseline and the fidelity earned by firing rules, for all five environments (below). DoorKey is the only environment where the fallback decides a substantial share of states. In the other four it decides at most 5% of states, and almost all fidelity comes from firing rules.
| Environment | States decided by fallback | Majority-action baseline | Total fidelity | Fidelity points from rules | Rule accuracy where rules fire |
|---|---|---|---|---|---|
| DoorKey-6x6 | 53.46 ± 0.54% | 53.37% | 99.85 ± 0.08% | 46.51 | 99.94% |
| Dynamic-Obs-5x5 | 0.01 ± 0.02% | 43.24% | 99.99 ± 0.02% | 99.99 | 100.00% |
| PixelCartPole | 4.95 ± 1.95% | 50.30% | 96.89 ± 0.96% | 94.70 | 99.64% |
| Pong | 0.17 ± 0.14% | 23.53% | 99.78 ± 0.13% | 99.77 | 99.95% |
| Boxing | 0.85 ± 0.45% | 26.19% | 98.40 ± 0.30% | 98.33 | 99.18% |
(Mean ± s.d. over three training seeds; same setting as the DoorKey table above. Total fidelity matches Table 2.)
- (ii) In the abstract and Section 4, we state that on DoorKey about half of the states are decided by the default action.
Q3: Does the joint-training stability argument hold for a value-based teacher?
The argument concerns the surrogate, not the teacher. LUCID only ever sees the teacher's frozen features and its arg-max action. Whether those come from a policy head (PPO) or from the arg-max over Q-values (DQN), the training objective is a cross-entropy on hard action labels; for the DQN teachers we additionally weight each state by the teacher's Q-value margin, so that near-ties in Q do not dominate the loss. What can change is the geometry of the features, so the question is empirical. We answer it by repeating the controls of Table 4 on the DQN teachers:
- two-stage: a reconstruction-only SAE, frozen before the DNF layer is trained;
- no SAE: the DNF layer applied directly to the CNN features.
LUCID was re-tuned for each DQN teacher over a wide hyperparameter range, and each control was re-tuned separately over its own grid, which also includes LUCID's selected configuration. The controls use exactly LUCID's data, train/test split, Q-margin sample weights, clause-bias cap and selection rule.
Hard fidelity (%) on LUCID's held-out 10% of the teacher rollouts, seed 42; literals in parentheses. "LUCID's configuration" is the control trained with LUCID's selected hyperparameters.
| Environment (DQN teacher) | LUCID | Two-stage, re-tuned | No SAE, re-tuned | Two-stage, LUCID's configuration | No SAE, LUCID's configuration |
|---|---|---|---|---|---|
| Dynamic-Obs-5x5 | 99.72 (10) | 98.06 (70) | 92.82 (50) | 88.32 (59) | 84.13 (72) |
| PixelCartPole | 96.88 (2) | 86.54 (25) | 85.27 (2,455) | 51.15 (2) | 71.37 (2) |
| Pong | 98.93 (52) | 77.37 (11,065) | 72.68 (11,386) | 51.48 (226) | 60.82 (243) |
| Boxing | 99.43 (178) | 73.93 (4,311) | 67.65 (61,237) | 65.83 (948) | 63.88 (783) |
| Enduro | 96.67 (255) | 59.89 (4,518) | 56.76 (54,843) | 54.14 (1,752) | 49.38 (1,650) |
The result carries over from PPO. No re-tuned control reaches LUCID's fidelity on any DQN teacher, and each needs more literals to get as close as it does. The gap is small on Dynamic-Obstacles (98.1% vs. 99.7%, with 7× more literals), 10 points on PixelCartPole, where at LUCID's own size of 2 literals the controls reach only 51–71%, 22 points on Pong, where the best control uses over 200× more literals, 25 points on Boxing (24× more literals) and 37 points on Enduro (18× more literals). Dynamic-Obstacles is the one case where a control comes close, and we say so in the paper.
One feature of the DQN teachers is worth noting. Their CNN features are sparser than those of the PPO teachers. Counting units that are constant over the training rollouts, 336 of the 512 CNN units are inactive for the DQN teacher on PixelCartPole, against 216 for PPO, and 300 against 152 on Boxing. LUCID therefore learns its dictionary from fewer live dimensions, which makes these settings a useful stress test for the sparsity argument.
On sparsity specifically. The two controls bracket the question from both sides. The no-SAE control removes the sparse dictionary altogether and applies the DNF layer to the dense CNN features; even after re-tuning it stays 7–40 points below LUCID's hard fidelity on the DQN teachers (5–28 points on the PPO teachers). The two-stage control keeps a sparse SAE but trains it for reconstruction only, and it also falls short of LUCID on every DQN teacher. A sparse dictionary is therefore needed, but sparsity alone does not keep the rules faithful after extraction; what does is training the dictionary jointly with the rule layer. We do not ablate the TopK operator separately (for example, with a dense autoencoder trained jointly), so we cannot say how much of the effect is due to sparsity per se. We state this distinction in the revision.
A correction to the numbers cited in the review. The reviewer's second strength cites the gap between the controls and LUCID in Table 4. Following reviewer sMWe (Q4), we re-tuned both controls per environment instead of reusing LUCID's hyperparameters, and re-tuning narrows that gap. Over three PPO seeds, the best re-tuned control now stays 5–28 points below LUCID's hard fidelity (92.1% vs. 99.9% on DoorKey, 89.1% vs. 99.99% on Dynamic-Obstacles, 91.6% vs. 96.9% on PixelCartPole, 72.2% vs. 99.8% on Pong, 72.8% vs. 98.4% on Boxing), and on Atari it still needs about 30–260× more literals. The conclusion that joint training is what keeps the rules faithful stands, but with a smaller margin than in the submission, and Table 4 now reports the re-tuned numbers.
We thank the reviewer for the careful reading and the precise questions. We answer them in the order of the review. A revised PDF has been uploaded, with all new or changed text marked in blue.
W1 & Q1: The divergence between alignment and causal efficacy rests on five environments with confounds
We agree with the reviewer, and we have revised both the claim and the evidence.
(a) Narrower claim. The paper already presents the divergence as "an observation, not a law" (Section 5 and Appendix E.8). Some sentences still read more strongly than that, in particular the abstract ("never hold together") and the conclusion. We rewrote them to state what was tested: the abstract now says that under our criteria the two "did not hold together in any of the five environments we tested", and the introduction and conclusion state that they did not hold together in any of the ten teacher–environment pairs we tested, including the DQN pairs below. The revision also states that this is an empirical finding about LUCID's dictionaries, not a general property of rule surrogates.
(b) More cases. Following the reviewers' suggestions, we repeated both audits with DQN teachers on five environments: the four original environments where DQN solves the task, plus Enduro, a new and visually richer Atari game. This gives five new teacher–environment pairs. On the four original environments the factor vocabularies and action spaces are unchanged while the teacher, and with it the features the dictionary is learned from, changes. Within the rebuttal period the DQN audits use one LUCID seed (42), so the causal verdict for DQN is "CI > 0 on seed 42" rather than "in all three seeds"; we mark these verdicts as provisional and will report three seeds in the camera-ready.
None of the new pairs is both aligned and causal, so the statement in its narrowed form holds for every pair we tested. We do not count the DQN pairs as strong new support for the divergence itself: with DQN on seed 42 neither property holds in any pair, so these pairs cannot show the two properties separating. What they do show is the paper's central point in its clearest form. LUCID's rules reach 96.7–99.7% hard fidelity on the DQN teachers (see Q3 and the response to DJUk), yet their concepts meet neither the alignment nor the causal criterion. Fidelity alone would have certified these surrogates; only the audits reveal the gap, which is why the paper recommends reporting fidelity, alignment and causal efficacy separately. The DQN pairs also show that, on the same environment with the same factors and action space, the verdicts change with the teacher: PixelCartPole, Pong and Boxing lose the causal verdict they have with PPO, and Dynamic-Obstacles loses its alignment verdict. We report these single-seed changes explicitly; they support reading the divergence as a property of particular teacher–dictionary pairs rather than of the environment alone, which is the narrower claim in (a).