- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of input sensitivity in differentiable logic gate networks (DLGN). Since each differentiable logic gate is only an intermediate continuous relaxation for optimization that is rounded to its closest binary gate after training, small perturbations may alter the binarized circuit substantially. To address this issue, the authors propose to add a Jacobian regularization term to the loss function. This Jacobian regularization term penalizes the norm of the partial derivative of layer with regard to its inputs from layer . The paper proposes two different Jacobian regularization terms: First, a Frobenius norm penalty, which uniformly weighs each entry of the Jacobian; secondly, a binarization-aware penalty, which weighs each entry by the proximity of the gate output to the binarization threshold . The contributions are mainly empirical. On the theoretical side, no guarantees or bounds on how the regularization term will affect the optimizer are presented. On the empirical side, the two regularization terms are compared against two other instances: a regularization-free baseline and a Jacobian regularization from prior work that applies a Jacobian regularization only on the last layer. The resulting models are optimized on several image classification datasets, ranging from MNIST over SVHN and CIFAR to Tiny-ImageNet. This choice is consistent with the evaluation setup that is used in previous work, which focused mainly on vision benchmarks. Their regularization terms achieve small but consistent improvements over these alternative methods. Finally, the authors show empirically that the norm of the Jacobian effectively decreases with their regularization terms to certify that the regularization penalty is active.
Critical strengths
- The exhibition is clear, accessible, and sound. My impression is that DLGNs are introduced in an understandable and comprehensive manner. The paper is mostly self-contained, all symbols are properly defined and only a few terms are left undefined and implicit. Results from previous work are also sufficiently cited. Mathematical derivations in the main paper and the parts of the appendix that I reviewed are correct.
- The empirical comparison adopts established benchmarks from previous work and present a few insights how variations in the regularization hyperparameter choices affects the downstream accuracy of the model. Also, the input signal perturbation of the threshold
Critical weaknesses
The design of the regularization term is not fully conclusive and imposes side effects that are not recognized or even argued against.
-
L2 regularization has a biased effect in this soft logic space that is different from its effect in usual real-valued, zero-centered vector spaces in standard architectures. Penalizing the partial derivatives that have the form drives all involved parameters towards 0. Not only the inputs (which are outputs of gates from the previous layer) might be inclined to shrink to , also the gate parameters are, which would lead to a constant function (cf. Equation 4). This penalty does thus not only act as a regularization away from perturbation-sensitive regions near the binarization threshold. Instead, it will steer all involved gates in the current and previous layers towards the constant 0. Again, in this soft-logic space, regularization is an asymmetric penalty and not a symmetric one as for zero-centered vector spaces. I would have expected from this paper of this kind to recognize this issue and to adjust the prior Jacobian regularization methods to this different feature space. The authors recognize the relation between the two (cf. "Therefore, the impact of a feature perturbation depends on the feature’s proximity to the binarization boundary.", line 239), but this insight is not reflected in their method. Instead, it seems that this asymmetric binarization bias that their method induces is not recognized (e.g. "BAJ does not directly encourage relaxed features to become binary", line 259).
-
The binarization-aware Jacobian regularization (BAJ) approaches this insight, but it still contains the asymmetric Jacobian regularization and only reweighs it differently. Moreover, it is not conclusive to me why the authors try to disentangle binarization from perturbation-sensitivity (e.g. "Thus, the binarization score only determines how strongly the sensitivity of each feature is penalized and does not itself push the feature toward a binary endpoint.", line 257), although they recognize the inevitable relation between them earlier in the paper (cf. "Therefore, the impact of a feature perturbation depends on the feature’s proximity to the binarization boundary.", line 239). That way, the paper leaves an unresolved impression that the binarization error from rounding the relaxed model could be separated from the minimization of perturbation-sensitivity. In particular, stopping gradients on the binarization-aware weights does not feel well-motivated.
Questions for authors
1a. Could you confirm that the gradients of their regularization penalty are computed both with respect to the inputs and the gate parameters ?
1b. Please provide more intuition in what way you regard the problem of minimizing discretization error and minimizing feature perturbation sensitivity as separable?
1c. Please elaborate why you think that BAJ would not encourage discretization? To my understanding, this regularization term drives gate outputs towards a constant gate, especially if the gate outputs are close to the rounding threshold.
2a. Why have the authors chosen stop-gradient in the BAJ formulation? Why would we enable gradient flow to both inputs and parameters on the one partial derivative of the product but not on the other?
3a. Could you please provide uncertainty measures for your stated accuracy results? The accuracy gap does often not seem substantial, and quantifying the uncertainty across seeds and different perturbation strengths would strongly support that this method exhibits a margin.
3b. I do not understand why you exclude MNIST from the perturbed classification accuracy in Table 2, which is in my opinion the most important comparison to substantiate the effectiveness of the regularization. I am also interested in the MNIST results since we see on SVHN that these new regularization methods can be even ineffective in increasing robustness against perturbed inputs.
3c. On the SVHN dataset, both the models that were trained with FJ and BAJ regularization fall below or on par to the baseline when tested on perturbed inputs. It hence seems to me that regularization was ineffective here. Do the authors have an explanation why that is?