acceptodds
Withdrawn from ICLR 2027

Stabilizing Logical Composition in Differentiable Logic Gate Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentiable logic gate networks (DLGNs) can be discretized into efficient hard logic circuits after training, but their intermediate relaxed Boolean features can be sensitive to input perturbations. This property is particularly relevant in DLGNs, where intermediate features serve as operands for subsequent logic gates and therefore directly participate in downstream logical composition. In this paper, we study convolutional DLGNs from the perspective of Boolean feature stability. We show that the bilinear interactions in relaxed logic gates induce feature-dependent local Jacobians, such that the local sensitivity of a gate varies with the values of its Boolean operands. Motivated by this analysis, we propose feature-level Jacobian regularization to stabilize selected intermediate relaxed Boolean features. We further introduce Binarization-Aware Jacobian regularization (BAJ), which assigns larger penalties to features near the binarization boundary, where perturbations are more likely to alter their Boolean interpretation. Experiments across image classification benchmarks show that the proposed regularizers improve classification performance and reduce the finite-perturbation sensitivity of intermediate Boolean features. Downstream analyses further show that feature stabilization affects subsequent logical computation. These results suggest that stabilizing intermediate Boolean operands is an effective approach to improving the stability of logical composition in convolutional DLGNs.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Differentiable Logic Gate Network, Robust Logical Composition, Jacobian Regularization, Representation Stability, Boolean Feature Stability, Differentiable Logic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WF1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of input sensitivity in differentiable logic gate networks (DLGN). Since each differentiable logic gate is only an intermediate continuous relaxation for optimization that is rounded to its closest binary gate after training, small perturbations may alter the binarized circuit substantially. To address this issue, the authors propose to add a Jacobian regularization term to the loss function. This Jacobian regularization term penalizes the norm of the partial derivative of layer i+1i+1 with regard to its inputs from layer ii. The paper proposes two different Jacobian regularization terms: First, a Frobenius norm penalty, which uniformly weighs each entry of the Jacobian; secondly, a binarization-aware penalty, which weighs each entry Dij=∂gi(x)∂xjD_{ij}=\frac{\partial g_i(x)}{\partial x_j} by the proximity of the gate output gi(x)g_i(x) to the binarization threshold 0.50.5. The contributions are mainly empirical. On the theoretical side, no guarantees or bounds on how the regularization term will affect the optimizer are presented. On the empirical side, the two regularization terms are compared against two other instances: a regularization-free baseline and a Jacobian regularization from prior work that applies a Jacobian regularization only on the last layer. The resulting models are optimized on several image classification datasets, ranging from MNIST over SVHN and CIFAR to Tiny-ImageNet. This choice is consistent with the evaluation setup that is used in previous work, which focused mainly on vision benchmarks. Their regularization terms achieve small but consistent improvements over these alternative methods. Finally, the authors show empirically that the norm of the Jacobian effectively decreases with their regularization terms to certify that the regularization penalty is active.

Critical strengths
  1. The exhibition is clear, accessible, and sound. My impression is that DLGNs are introduced in an understandable and comprehensive manner. The paper is mostly self-contained, all symbols are properly defined and only a few terms are left undefined and implicit. Results from previous work are also sufficiently cited. Mathematical derivations in the main paper and the parts of the appendix that I reviewed are correct.
  2. The empirical comparison adopts established benchmarks from previous work and present a few insights how variations in the regularization hyperparameter choices affects the downstream accuracy of the model. Also, the input signal perturbation of the threshold
Critical weaknesses

The design of the regularization term is not fully conclusive and imposes side effects that are not recognized or even argued against.

  1. L2 regularization has a biased effect in this soft logic space that is different from its effect in usual real-valued, zero-centered vector spaces in standard architectures. Penalizing the partial derivatives that have the form w1+w3b,w2+w3a(cf. Equation 7)w_{1} + w_{3} b, \quad w_2 + w_3 a \quad \text{(cf. Equation 7)} drives all involved parameters towards 0. Not only the inputs a,ba,b (which are outputs of gates from the previous layer) might be inclined to shrink to 00, also the gate parameters w1,w2,w3w_{1},w_2,w_3 are, which would lead to a constant function g(a,b)=w0g(a,b)=w_0 (cf. Equation 4). This penalty does thus not only act as a regularization away from perturbation-sensitive regions near the binarization threshold. Instead, it will steer all involved gates in the current and previous layers towards the constant 0. Again, in this soft-logic space, regularization is an asymmetric penalty and not a symmetric one as for zero-centered vector spaces. I would have expected from this paper of this kind to recognize this issue and to adjust the prior Jacobian regularization methods to this different feature space. The authors recognize the relation between the two (cf. "Therefore, the impact of a feature perturbation depends on the feature’s proximity to the binarization boundary.", line 239), but this insight is not reflected in their method. Instead, it seems that this asymmetric binarization bias that their method induces is not recognized (e.g. "BAJ does not directly encourage relaxed features to become binary", line 259).

  2. The binarization-aware Jacobian regularization (BAJ) approaches this insight, but it still contains the asymmetric Jacobian regularization and only reweighs it differently. Moreover, it is not conclusive to me why the authors try to disentangle binarization from perturbation-sensitivity (e.g. "Thus, the binarization score only determines how strongly the sensitivity of each feature is penalized and does not itself push the feature toward a binary endpoint.", line 257), although they recognize the inevitable relation between them earlier in the paper (cf. "Therefore, the impact of a feature perturbation depends on the feature’s proximity to the binarization boundary.", line 239). That way, the paper leaves an unresolved impression that the binarization error from rounding the relaxed model could be separated from the minimization of perturbation-sensitivity. In particular, stopping gradients on the binarization-aware weights does not feel well-motivated.

Questions for authors

1a. Could you confirm that the gradients of their regularization penalty are computed both with respect to the inputs a,ba,b and the gate parameters ww?

1b. Please provide more intuition in what way you regard the problem of minimizing discretization error and minimizing feature perturbation sensitivity as separable?

1c. Please elaborate why you think that BAJ would not encourage discretization? To my understanding, this regularization term drives gate outputs towards a constant gate, especially if the gate outputs are close to the rounding threshold.

2a. Why have the authors chosen stop-gradient in the BAJ formulation? Why would we enable gradient flow to both inputs and parameters on the one partial derivative of the product but not on the other?

3a. Could you please provide uncertainty measures for your stated accuracy results? The accuracy gap does often not seem substantial, and quantifying the uncertainty across seeds and different perturbation strengths would strongly support that this method exhibits a margin.

3b. I do not understand why you exclude MNIST from the perturbed classification accuracy in Table 2, which is in my opinion the most important comparison to substantiate the effectiveness of the regularization. I am also interested in the MNIST results since we see on SVHN that these new regularization methods can be even ineffective in increasing robustness against perturbed inputs.

3c. On the SVHN dataset, both the models that were trained with FJ and BAJ regularization fall below or on par to the baseline when tested on perturbed inputs. It hence seems to me that regularization was ineffective here. Do the authors have an explanation why that is?

Reviewer_FawoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the stability of intermediate relaxed Boolean features in Convolutional Differentiable Logic Gate Networks (CLGNs). Since these features are reused as inputs to later logic gates, instability at an intermediate layer can propagate through subsequent logical computation. Based on the feature-dependent Jacobians induced by relaxed logic gates, the paper proposes Feature-Jacobian (FJ) regularization and a binarization-aware variant, BAJ. FJ penalizes input sensitivity of selected intermediate features, while BAJ places more weight on features close to the binarization boundary. Experiments on five image-classification datasets show consistent improvements in clean accuracy, and the CIFAR-10 analysis shows substantially reduced intermediate feature sensitivity. The robustness-related results are less consistent across datasets.

The paper is technically interesting and well motivated, and the CIFAR-10 analysis provides useful evidence that the proposed regularizers affect the intended intermediate representations. However, the current empirical evidence is not fully convincing. The lack of multi-run statistics, unclear hyperparameter selection, dataset-dependent robustness results, and CIFAR-10-only mechanistic analysis make it difficult to judge the robustness and generality of the reported gains.

Critical strengths

The paper targets a meaningful property of differentiable logic networks. Intermediate relaxed Boolean features are directly reused by later logical operations, so stabilizing them is well motivated and more specific to the model structure than regularizing only the final output.

The evaluation covers five datasets and includes both the unregularized CLGN baseline and output-level Jacobian regularization. FJ and BAJ improve reported clean accuracy on all five datasets.

The CIFAR-10 analysis is also useful. Both methods reduce sensitivity at intermediate stages, including layers downstream of the directly regularized representation. The downstream-influence results suggest that regularization does not simply suppress the intermediate representation; instead, its influence becomes more concentrated across feature dimensions.

Critical weaknesses

The main issue is the lack of multi-run statistics. Several gains are small: for example, BAJ improves over the baseline by 0.12 points on MNIST, 0.50 on SVHN, and 0.77 on CIFAR-100. Table 1 reports single values without standard deviations, confidence intervals, or a clearly stated multi-seed protocol. It is therefore hard to tell whether these differences are larger than normal training variance.

The perturbation results are also not consistent across datasets. On SVHN, FJ reduces perturbed accuracy from 65.85% to 64.18%, while BAJ is essentially unchanged from the baseline. Neither method improves logit sensitivity there. This weakens any general claim that reducing intermediate feature sensitivity necessarily improves output robustness.

The hyperparameter-selection procedure is not sufficiently clear. The CIFAR-10 ablation selects the BAJ coefficient with the best accuracy, but it is unclear whether this is based on a validation set or the test set. The paper should also report the regularization strengths used for FJ, BAJ, and OJ on each dataset.

Most of the mechanistic analysis is limited to CIFAR-10. Given the different behavior on SVHN, it would be useful to know whether intermediate features are also stabilized there.

Finally, FJ and BAJ require additional gradient computations during training, but the paper does not report training-time or memory overhead. Since inference is unchanged, this is mainly a training-cost issue, but it should still be quantified.

Questions for authors

How many independent runs were used for Table 1? Could the authors report mean and standard deviation over multiple seeds?

How were the regularization strengths for FJ, BAJ, and OJ selected on each dataset? Was a separate validation set used?

What is the training-time and memory overhead of FJ and BAJ relative to the baseline?

On CIFAR-10, FJ gives stronger feature stabilization than BAJ, but BAJ achieves better clean and perturbed accuracy. How do the authors explain this trade-off?

Could the authors provide intermediate-sensitivity analysis on another dataset, especially SVHN, where the robustness results differ from CIFAR-10?

Reviewer_kGRjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper applies feature-level regularization to convolutional logic gate networks. The authors argue that noise in the input can destabilize intermediate features and thereby degrade final image classification performance. By applying feature-level and binarization-aware Jacobian regularization at selected layers, the proposed method improves accuracy over the baseline on the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.

Critical strengths

none

Critical weaknesses
  • No explanation of why the method works in Boolean networks. The paper applies existing Jacobian regularization to logic gate networks, but it explains neither the source of the feature instability it targets nor why regularization is effective in this setting. The only justification, the bilinear interaction term in the Introduction, is unsupported by derivation. Since the regularization technique itself is not new, the contribution depends on showing why it suits Boolean networks. Without this analysis, the paper reduces to an empirical application of an existing method.

  • Insufficiently rigorous experiments. Key hyperparameters (σ and the regularized layer k*) are chosen without justification. The choice of k* = Conv2 contradicts the authors’ own sensitivity analysis, which identifies Conv4. It is also unclear whether the results come from a single run. The paper’s evidence is its accuracy gain over the baseline. If that gain depends on unexplained choices and may lie within run-to-run variance, the central claim is not reliably supported.

Questions for authors

Q1. Missing references and definitions. Please add references for Equation (1), which we could not find in the DLGN paper or the other cited works; if it is the authors' own formulation, state this explicitly. Please also cite the source for the discretization step in Section 2.1 (line 84), define h^(l)(x), and explain in what sense the “relaxed Boolean features” (Section 3, line 144) are relaxed.

Q2. Soft gate distribution. What is "the soft gate distribution" in Equation (4), and how does it differ from Equation (1)?

Q3. Source of feature instability. Section 4 and Figure 1 describe feature instability but do not explain why it arises. Is there a derivation or theoretical argument showing that the intermediate relaxed Boolean features are unstable?

Q4. Effectiveness of the proposed regularization. Please justify, beyond empirical results, why Feature-Jacobian Regularization and Binarization-Aware Jacobian Regularization are effective in Boolean networks. In particular, clarify where and how binarization is applied during training.

Q5. Experimental design. (a) Why is the perturbation scale set to σ = 8/255, and how sensitive are the results to this choice? (b) Why are the MNIST results in Table 1 omitted from Table 2? (c) How is the layer k* selected? Figure 3 shows Conv4 has the highest sensitivity, yet Conv2 is chosen. (d) Are the results from a single run? If so, please report the mean and standard deviation over multiple seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.