acceptodds
Under review as a conference paper at ICLR 2027

Towards Co-Discovery By Extracting A Symbolic Representation From A Network Layer: Complexity, Algorithms and Experiments

Abstract

Deep learned models (DLMs) have been shown to be useful in a wide variety of situations. However, most XAI methods provide simple instance-level heatmap-style explanations that neither reveal the computation performed by the deep learner nor allow a human to modify the computation. Given that most DLMs are comprised of simple computational units, we explore extracting out a symbolic representation of the computation performed by the model as a formula in disjunctive normal form (DNF). We then show how the DNF can be used not only for explanation but also for simplification based on fairness/complexity/accuracy and even model editing. To achieve a symbolic representation, we restrict the layer's input to binary inputs often found in graphical data and some architectures. We explore three core co-discovery problems: extraction, simplification, and editing, showing complexity results, algorithms, and experimental results for each.

Primary area:
interpretability and explainable AI
Keywords:
XAI, Editing, Neurosymbolic

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MMCUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes representing each neuron of a fully connected layer with binary inputs as a logic rule in disjunctive normal form (DNF) over its inputs. The approach is divided into three parts, each accompanied by complexity results. First, extraction: since enumerating all conjunctive terms is intractable, a Gibbs sampler estimates the data distribution of the input configurations and is used to extract a subset of highly probable terms. Second, simplification: the extracted terms are reduced through product term removal, which is solved by dynamic programming in the special case where the cost of a term is its number of literals. Third, editing: a linear program finds the weights that realize a given modification of the extracted rule. The experiments show that the sampler extracts more probable and more accurate terms than random selection, and that the simplification step reduces the complexity of the rules. On the ADNI dataset, edits based on prior domain rules (symmetry, triangle closing, connectivity), applied automatically to simulate a human expert, increase the accuracy of the network once the edited weights are reinserted.

Critical strengths
  • An interesting approach to make rule extraction from the activation of a neuron tractable. Enumerating the logic rules equivalent to the activation of a neuron is intractable, and the paper addresses this with a sampling-based approach that extracts a subset of highly probable terms. This matters because rule extraction is a long-standing and difficult problem for interpretability.
  • The extract–simplify–edit loop. The paper goes beyond explanation by proposing to simplify the extracted rules and to edit them, reinserting the modified rule into the network as new weights. This direction is related in spirit to model editing in mechanistic interpretability and is a very important matter in current deep learning xai and safety literature.
Critical weaknesses

1. The extracted DNF does not faithfully represent, nor allow editing of, a generic neuron.

  • The DNF only captures whether the unit is active, i.e. 1[w⊤x+b>0]\mathbb{1}[w^\top x + b > 0], and not the magnitude of its activation, which is what the next layer receives with a ReLU. To the best of my understanding, the DNF is therefore not a faithful representation of a neuron even in the first layer with binary inputs. This would only hold if activations were (nearly) binary, e.g. with saturated sigmoids or binarized activations.
  • The approach does not apply to generic networks, only to very specific ones: binary inputs, non-negative weights and only negative biases (positive thresholds). As the experiments confirm (Sec. 7.1, l. 377–381; Sec. 7.2, l. 410–411), this requires constraining the training of the network, which becomes monotone in every input and thus has strongly limited representational capacity, and it can only be applied to the input layer.
  • Some DNFs cannot be represented by a single threshold unit: for instance, no weights and threshold realize (x1∧x2)∨(x3∧x4)(x_1 \wedge x_2) \vee (x_3 \wedge x_4). As a consequence, edited or simplified DNFs may not be reinsertable into the network.

2. The experimental validation and related work fall below the standard expected at ICLR.

  • No other methods are compared. Rule learners such as BRL, association rule mining or decision trees could be used to explain a layer or the whole model, or to solve the same tasks directly. In that case editing could be performed directly on the model, by changing a rule or a branch, and non-binary outputs could also be explained, e.g. with regression trees. The only baseline is random term selection, and the authors themselves note that the result in Table 2 is expected by construction (l. 390). The effectiveness of the simplification step is also hard to assess, since the caption of Table 3 contradicts both its numbers and the text (l. 396).
  • No standard deviations are reported in Tables 2–4, although the results are averaged over 500 runs.
  • The related work misses several aspects: rule extraction from neural networks is discussed only minimally; association rule mining, ILP and symbolic regression (e.g., genetic programming-based approaches and, more recently, Kolmogorov–Arnold Networks) are not discussed; and contemporary model editing approaches, e.g. based on mechanistic interpretability, are completely missing.

Minor remarks (presentation).

  • The bias is treated as a threshold (Def. 3.1, b>0b > 0 must be exceeded), which implies that the actual bias must be negative, whereas in the standard formulation the bias contributes additively to the activation. The introductory example (l. 64) should accordingly use b=−1.9b = -1.9.
  • Several acronyms are used without being introduced or inconsistently: DLM, DL, PDL.
  • Algorithm 1 is reported but not explained in the text, and some of its variables (e.g., BurnIn) are never introduced.
Questions for authors
  1. Why did you not compare your method against any other approach for explaining a neural model (e.g., rule extraction methods, decision trees, BRL, association rule mining)? Such a comparison is necessary to assess the benefits of the proposed extraction and simplification.
  2. How exactly are the DNF terms obtained from Algorithm 1? The algorithm returns a distribution over input configurations: how many samples are drawn (values of NumSamples and BurnIn), and how are the “100 product terms” used in Tables 2–3 selected from this distribution?
  3. What happens when an edited or simplified DNF cannot be realized by a single threshold unit, e.g. (x1∧x2)∨(x3∧x4)(x_1 \wedge x_2) \vee (x_3 \wedge x_4), so that the LP in Eq. (1) is infeasible? More generally, how does editing behave when the extracted DNF is a poor approximation of the original unit?
  4. Could you report standard deviations for the results in Tables 2–4?
Reviewer_CfQSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a symbolic model analysis framework for human-AI co-discovery. It aims to extract disjunctive normal form (DNF) rules from ReLU network layers with binary inputs and non-negative weights, allow humans to simplify and edit these rules, and finally convert the modified rules back into neural network parameters. The experiments are conducted on three UCI datasets and ADNI brain connectivity data. The results show that the Gibbs sampler is more likely than random sampling to find high-frequency rules, while dynamic programming can reduce rule complexity and improve coverage. In the ADNI experiments, the authors simulate domain-expert editing using three types of rules: left-right symmetry, triangle closure, and subgraph connectivity. The edited model achieves the highest average accuracy on both the AD and MCI classification tasks.

Critical strengths
  1. Rather than merely extracting post-hoc explanations, the paper attempts to establish a complete loop of rule extraction, rule simplification, human editing, and writing the edited rules back into the network. This framework places greater emphasis on the actionability of the explanations.
  2. The paper discusses the computational complexity of the extraction, simplification, and editing problems, and proposes solutions including Gibbs sampling, dynamic programming, approximation algorithms, and linear programming for problems that are difficult to solve exactly.
  3. The paper is generally well organized. The Extraction, Simplification, and Editing sections broadly correspond to the relevant theoretical results and experimental questions, making the overall workflow easy to follow.
Critical weaknesses
  1. The current framework relies on binary inputs, non-negative weights, and ReLU units, and the experiments focus primarily on the first layer. This setting corresponds to monotone Boolean threshold functions and remains far removed from general neural networks with continuous inputs, negative weights, and deep representations.
  2. The DNF appears to represent only the activation state of a ReLU rather than its complete output. A DNF can represent whether a weighted sum exceeds a threshold, but it cannot represent the continuous value produced after ReLU activation. Therefore, the paper's claim that the DNF performs exactly the same computation as the neuron may not be sufficiently precise, and it is also unclear how the continuous activation magnitudes are preserved when the network is reconstructed.
  3. The evidence for human-AI co-discovery and interpretability is limited. In the ADNI experiments, an individual conjunction after simplification still contains hundreds of variables on average. It is questionable whether such rules can be effectively understood and edited by experts.
Questions for authors
  1. Does the DNF represent whether a ReLU is activated, or does it represent the complete continuous output of the ReLU? If it represents only the activation state, how do the authors ensure that the downstream layers receive activation magnitudes consistent with those of the original model after the weights are reconstructed from the DNF?
  2. During the Gibbs sampler updates, how do the authors ensure that every sampled state is a valid or minimally valid product term for the target neuron? Why does Algorithm 1 no longer explicitly use the neuron's weights and threshold after initialization?
  3. What exactly does the accuracy reported in Table 3 refer to: product-term accuracy, DNF accuracy, or the test accuracy of the complete model? Are rule extraction, simplification, and final evaluation all performed strictly within each cross-validation fold to avoid data leakage?
  4. After replacing the first-layer weights, are the downstream layers frozen or fine-tuned? If frozen, why should they remain calibrated to the modified activations? If fine-tuned, how much of the performance gain comes from retraining rather than symbolic editing?
Reviewer_kjtxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes using disjunctive normal form (DNF) to extract symbolic representations from deep learning models, with the goal of revealing their internal computations and allowing humans to modify them. However, the methodology is limited by the requirement that the inputs to the analyzed layer must be binary.

Critical strengths
  • The authors evaluate their methodology on four different datasets;
  • The authors describe the algorithms used in their methodology, which helps clarify the proposed approach and supports reproducibility.
Critical weaknesses
  1. Motivation: The authors motivate their work by discussing classical XAI methods, such as LIME, which do not aim to explain a model’s internal mechanisms. It would be more appropriate to motivate the work by directly discussing existing methods in mechanistic interpretability and explaining how the proposed approach differs from or improves upon them.

  2. Methodology: The training constraints are not clearly defined. Are all layers binary, or does this apply only to specific layers? How is the model trained under these constraints? Although some tasks involve categorical data, converting these variables into one-hot encodings can substantially increase the input dimensionality. I am not convinced that binary representations are necessarily more interpretable to humans, as this transformation may remove semantic information and increase the number of features that need to be analyzed. Do the authors provide evidence to support this claim? Could they provide an example of a human intervention that becomes easier with this approach?Furthermore, the intuition behind the definitions of valid and minimally valid product terms requires further explanation. The proposed simplification appears to rely on special cases that may limit its applicability to more general settings. Finally, it is unclear how humans are expected to edit the extracted rules by deleting literals or modifying constraints. The authors should clarify this process and provide a concrete example to illustrate how such interventions are performed in practice.

  3. Intrinsic Explanation: Is the proposed method an intrinsic explanation approach, in which the model is trained under specific constraints to make its internal mechanisms more interpretable? If so, this should be clearly stated from the beginning, as I initially expected a method for explaining and editing existing models. Furthermore, the authors should compare the performance of the constrained model with that of standard models on the same tasks to demonstrate whether the interpretability constraints lead to a significant loss in predictive accuracy.

  4. Experiments and Results: The authors should cite and describe the datasets used in the experiments (Mushroom, Breast Cancer, and Healthy Aging) to provide sufficient context for interpreting the results. The statement that “The simplification method yields slightly higher accuracy, but with significantly greater coverage and complexity” also appears inconsistent with the table, which reports lower complexity for the simplification method. The authors should clarify this. The paper would also benefit from a clearer discussion of the main conclusions from the experiments. How suitable is the proposed method for real-world applications, and what limitations might arise when applying it to high-dimensional data?

Questions for authors

I have mentioned the questions above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.