- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes representing each neuron of a fully connected layer with binary inputs as a logic rule in disjunctive normal form (DNF) over its inputs. The approach is divided into three parts, each accompanied by complexity results. First, extraction: since enumerating all conjunctive terms is intractable, a Gibbs sampler estimates the data distribution of the input configurations and is used to extract a subset of highly probable terms. Second, simplification: the extracted terms are reduced through product term removal, which is solved by dynamic programming in the special case where the cost of a term is its number of literals. Third, editing: a linear program finds the weights that realize a given modification of the extracted rule. The experiments show that the sampler extracts more probable and more accurate terms than random selection, and that the simplification step reduces the complexity of the rules. On the ADNI dataset, edits based on prior domain rules (symmetry, triangle closing, connectivity), applied automatically to simulate a human expert, increase the accuracy of the network once the edited weights are reinserted.
Critical strengths
- An interesting approach to make rule extraction from the activation of a neuron tractable. Enumerating the logic rules equivalent to the activation of a neuron is intractable, and the paper addresses this with a sampling-based approach that extracts a subset of highly probable terms. This matters because rule extraction is a long-standing and difficult problem for interpretability.
- The extract–simplify–edit loop. The paper goes beyond explanation by proposing to simplify the extracted rules and to edit them, reinserting the modified rule into the network as new weights. This direction is related in spirit to model editing in mechanistic interpretability and is a very important matter in current deep learning xai and safety literature.
Critical weaknesses
1. The extracted DNF does not faithfully represent, nor allow editing of, a generic neuron.
- The DNF only captures whether the unit is active, i.e. , and not the magnitude of its activation, which is what the next layer receives with a ReLU. To the best of my understanding, the DNF is therefore not a faithful representation of a neuron even in the first layer with binary inputs. This would only hold if activations were (nearly) binary, e.g. with saturated sigmoids or binarized activations.
- The approach does not apply to generic networks, only to very specific ones: binary inputs, non-negative weights and only negative biases (positive thresholds). As the experiments confirm (Sec. 7.1, l. 377–381; Sec. 7.2, l. 410–411), this requires constraining the training of the network, which becomes monotone in every input and thus has strongly limited representational capacity, and it can only be applied to the input layer.
- Some DNFs cannot be represented by a single threshold unit: for instance, no weights and threshold realize . As a consequence, edited or simplified DNFs may not be reinsertable into the network.
2. The experimental validation and related work fall below the standard expected at ICLR.
- No other methods are compared. Rule learners such as BRL, association rule mining or decision trees could be used to explain a layer or the whole model, or to solve the same tasks directly. In that case editing could be performed directly on the model, by changing a rule or a branch, and non-binary outputs could also be explained, e.g. with regression trees. The only baseline is random term selection, and the authors themselves note that the result in Table 2 is expected by construction (l. 390). The effectiveness of the simplification step is also hard to assess, since the caption of Table 3 contradicts both its numbers and the text (l. 396).
- No standard deviations are reported in Tables 2–4, although the results are averaged over 500 runs.
- The related work misses several aspects: rule extraction from neural networks is discussed only minimally; association rule mining, ILP and symbolic regression (e.g., genetic programming-based approaches and, more recently, Kolmogorov–Arnold Networks) are not discussed; and contemporary model editing approaches, e.g. based on mechanistic interpretability, are completely missing.
Minor remarks (presentation).
- The bias is treated as a threshold (Def. 3.1, must be exceeded), which implies that the actual bias must be negative, whereas in the standard formulation the bias contributes additively to the activation. The introductory example (l. 64) should accordingly use .
- Several acronyms are used without being introduced or inconsistently: DLM, DL, PDL.
- Algorithm 1 is reported but not explained in the text, and some of its variables (e.g., BurnIn) are never introduced.
Questions for authors
- Why did you not compare your method against any other approach for explaining a neural model (e.g., rule extraction methods, decision trees, BRL, association rule mining)? Such a comparison is necessary to assess the benefits of the proposed extraction and simplification.
- How exactly are the DNF terms obtained from Algorithm 1? The algorithm returns a distribution over input configurations: how many samples are drawn (values of NumSamples and BurnIn), and how are the “100 product terms” used in Tables 2–3 selected from this distribution?
- What happens when an edited or simplified DNF cannot be realized by a single threshold unit, e.g. , so that the LP in Eq. (1) is infeasible? More generally, how does editing behave when the extracted DNF is a poor approximation of the original unit?
- Could you report standard deviations for the results in Tables 2–4?