acceptodds
Under review as a conference paper at ICLR 2027

Fitting Multilinear Polynomials for Logic Gate Networks

Abstract

We study learnable logic gate networks that stack layers of 2-input Boolean gates to build combinational circuits. Every 2-input gate has a unique multilinear polynomial with 4 coefficients, so the 16 Boolean gates form a codebook of prototypes in a 4-dimensional space, reducing training to a vector-quantization problem. The baseline method, Soft-Mix, learns a 16-dimensional softmax over gate identities, but the codebook has rank 4: 11 of 15 simplex directions carry nullspace gradient, and at uniform initialization the backward signal vanishes exactly. We prove that no affine product reparameterization fixes the resulting interaction-coefficient starvation under STE, and show that the covariance Jacobian of soft-VQ selection bypasses it by coupling the starved coefficient to the always-active constant channel. Working in the 4-dimensional polynomial space reduces each neuron from 16 to 4 parameters. On seven datasets, with every accuracy measured on the deployed strict-Boolean (zero-multiply-accumulate) circuit, our primary method, Multilinear-CovJac, matches or exceeds 16-parameter Soft-Mix on every dataset, and its advantage over STE grows monotonically with interaction demand. At depth, Soft-Mix collapses (pp on CIFAR-10 at 12 layers) while CovJac holds (pp on CIFAR-10, stable on MNIST).

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
efficient inference, straight-through estimator, vector quantization, multilinear polynomial.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SghtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The submission studies the training of logic gate networks (LGNs). It makes some technical observations, in particular that the originally proposed 16-gate softmax formulation has degenerate gradients, and it proposes two new techniques based on quantization to the nearest codebook entry, which require fewer parameters per gate than the softmax (though not fewer than related works). Experiments in typical LGN benchmarks show improvements over classic baselines, but a apples-to-apples comparison to recent alternative training techniques is missing.

Critical strengths
  • The manuscript addresses an important problem in the context of logic-gate networks: improving their training.

  • The insight of degeneracy in the optimization are interesting and relevant.

  • The proposed method makes sense and seem to lead to improvements.

Critical weaknesses
  • The technical depth and breadth of the result are limited. The problems of training in the 16-gate SoftMix representation have been reported before, even though less formally, and most recent LGN works do not use plain SoftMix for training anymore. While the reported insights follow from rather elementary observations and computations. While they are good to know, they might be better batch to topical workshop. The proposed new methods are ad-hoc suggestion, that make sense, but do not stand out compared to methods suggested previously in the literature.

  • The submission lacks comparison to important related work. Conceptually, a comparison to CAGE (Kim, ICML 2026) would be important, as that starts from the same observations and also proposes an alternative way of training. Experimentally, the comparison just to plain baselines (SoftMix, Gumbel-ST) is insufficient, and other recently proposals should be included, at least LightDLGN (Ruttgers etal, ICLR 2026) and CAGE.

  • The writing is not clear enough for publication. It relies too much on technical terms and implicit assumptions, while also arguing with concept that are not truly defined (e.g. "interaction coefficient starvation").

Questions for authors

I do not have questions that could materially affect my assessment.

3 comments
AuthorsOpenReview ↗

(Part 1/2)

We thank the reviewer for the detailed comments and for acknowledging the strengths: "an important problem" (Strength 1), insights that "are interesting and relevant" (Strength 2), and a method that "makes sense" (Strength 3). On the missing "comparison to recent alternative training techniques": the revised PDF cites the recent methods, which differ from ours in kind, and WARP-LNN, run in our harness, is comparable to the methods in our tables (see [W2]). Our 4-vs-16 reduction is relative to the 16-gate softmax, not a claim against these works. On technical depth, Proposition 7 is an impossibility result over a design space, not an elementary observation (see [W1]); on clarity, see [W3]. Line, table, and appendix numbers refer to the submitted PDF unless marked "revised". In the revised PDF, revised and added text is shown in color; new references are listed in the bibliography.

[W2] How does this work differ from CAGE, LightDLGN (IWP), and other recent methods? Our method differs from these in kind, not only in basis. Soft-Mix, Gumbel-ST, and architecture-search relaxations learn a probability over the 16 gates. CAGE (Kim, ICML 2026) keeps that 16-way gate and adapts its backward temperature for robustness; in its accuracy-versus-temperature figure (Fig 2b), it differs from Gumbel-ST only at low temperature. At temperature 1, the Gumbel temperature of our Gumbel-ST baseline, the two reach similar accuracy (about 98% on MNIST), so our tables include Gumbel-ST as the closest stand-in. LightDLGN and WARP-LNN learn, through an output activation, the probability that each truth-table entry is 1, and use residual initialization at depth. We instead fit the gate: four multilinear coefficients, with the gate selected by its distance to the 16 Boolean gates, softly in training (soft vector quantization, Eq. 4) and as the nearest gate at deployment (Eq. 2), with no output activation or added loss. The difference matters. Because complementary gates have opposite input coefficients (Lemma 1), a free probability over the 16 gates, as in Soft-Mix, cancels its own input gradient at uniform (Proposition 2) and shrinks it near uniform. Weights set by distance from four fitted coefficients are never uniform (Lemma 3), and the exact gradient of the fit reaches the interaction coefficient on every sample (Proposition 5). We claim this principle and its analysis, not higher accuracy than these methods (see also our reply to NCTb [related work]). We thank the reviewer for pointing to CAGE and LightDLGN; the further recent methods are discussed in our reply to NCTb [related work].

Where public code exists, we ran it; for LightDLGN, whose code is unavailable, we compare the bases. WARP-LNN, run from the public torchlogix implementation in our MNIST harness (new; L=6, same width, batch, and learning rate; 3 seeds), reaches 98.09 ±\pm 0.02 with residual initialization and 98.03 ±\pm 0.01 with random initialization, comparable to the methods in our tables. On MNIST, CovJac first reaches 98% at 2k to 3k iterations (Table 1 runs), and WARP-LNN at 18k to 23k (these runs). LightDLGN's linked repository is unavailable, so we compare bases in one harness under straight-through training: the corner basis reaches 80.84, against 98.09 for Multilinear-STE, our canonical basis (Table 22; details in our reply to NCTb [corner basis]). These numbers describe the bases under our training, not LightDLGN's method, which trains differently.

[W1] Are the results elementary? They are not: Proposition 7 is an impossibility result over a whole design space, showing that no affine product basis attains full coverage, gradient coherence, and zero straight-through bias at once (proved in App A.6). Together with Proposition 5 and Lemma 3, it predicts, before any experiment, which methods starve and which collapse at depth, and the experiments follow that prediction. The earlier, less formal reports of the Soft-Mix problem are credited (Sec 2), and none contains this impossibility result. Nor is plain Soft-Mix our only baseline: Gumbel-ST, the stochastic hard selection used in recent DLGN training (Kim 2023; Yousefi et al. 2025), is in every main and depth table.

The method follows from the same analysis and is not ad hoc. There is one proposal, Multilinear-CovJac, trained by the exact gradient of the soft-VQ forward (Eq. 5), with Multilinear-STE as the ablation anchor (L383-384); [W2] describes how it differs from recent methods.

[W3] Are the terms defined? The term the reviewer names stands for one statement, made in Sec 3.1: under straight-through training, the interaction coefficient "receives gradient on only 1 in 4 samples" (under balanced inputs, Lemma 6). Its earlier uses, in the abstract and the contribution list, carry the same meaning. Our reply to NCTb [presentation] lists the clarity changes.

AuthorsOpenReview ↗

(Part 2/2)

[ethics] Is the Adult dataset appropriate? We use Adult only for comparability with prior DLGN work, not to study fairness. The revised Ethics statement says so and cites Retiring Adult (Ding et al. 2021) for its limitations. We also ran ACS Income, the replacement that Retiring Adult proposes (new, revised App E.14; 3 seeds): CovJac reaches 81.95 ±\pm 0.03, against 81.85 ±\pm 0.05 for Soft-Mix, 81.81 ±\pm 0.02 for Multilinear-STE, and 81.76 ±\pm 0.05 for Gumbel-ST, so the four methods tie within 0.2pp, as they do on Adult (within 0.3pp).

Thank you for your response. I have no questions at this point and will maintain my score while awaiting the upcoming discussion with the other reviewers and the area chair.

Reviewer_9qrhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training of differentiable logic gate networks by representing the 16 two-input Boolean gates in a four-dimensional multilinear coefficient space. It proposes Multilinear-CovJac, which uses soft vector quantization and a covariance-Jacobian gradient mechanism to improve optimization, particularly for interaction-related coefficients. The method is evaluated on several classification datasets and compared with Soft-Mix, Gumbel-ST, and a multilinear STE variant. The deployed models remain strict Boolean gate networks.

Critical strengths

The paper gives a clear analysis of optimization difficulties in differentiable logic gate networks and provides a coherent training approach that helps to reduce the learned representation from 16 to 4 parameters per neuron.

The experimental evaluation is relatively extensive and includes several datasets, depth studies, and diagnostic experiments. In particular, the depth results provide useful evidence that the proposed training approach can improve stability while preserving a Boolean circuit at deployment.

Critical weaknesses

My main concern is the significance of the contribution relative to the closest prior work. The reduction from 16 gate parameters to a four-parameter representation is not itself new, and prior work has already identified the complementary-gate gradient-cancellation problem. The main new contribution is therefore the particular multilinear representation together with the soft-VQ/CovJac training mechanism. While this is technically interesting and the experimental results are generally positive, I find the step from the existing literature relatively incremental for ICLR.

A second concern is the trade-off introduced by the proposed training method. One of the main motivations for logic-gate networks is their direct Boolean computation and very simple hardware realization. The final CovJac-trained model retains this advantage at deployment, but the training procedure itself operates in a continuous arithmetic domain, requiring distance calculations over the gate codebook, softmax weighting, weighted coefficient combinations, and Jacobian-based optimization. Thus, the proposed method moves training away from the logic-native character that motivates this class of models. The paper provides analytical operation-count estimates, but does not demonstrate how this training procedure maps to dedicated logic/FPGA hardware or provide measured training efficiency. I would have liked a clearer discussion of this trade-off and of whether the efficiency motivation is intended primarily for deployment rather than training.

Questions for authors

The paper describes the Multilinear-STE versus Multilinear-CovJac comparison as isolating what the CovJac gradient mechanism adds. However, the two methods also use different training forward passes: M-STE hard-snaps to a gate, whereas CovJac uses a soft-VQ mixture. Therefore, the comparison appears to isolate the complete soft-VQ/CovJac training mechanism rather than the covariance-Jacobian effect alone. Can the authors clarify how much of the observed accuracy improvement can be attributed specifically to the covariance coupling, as opposed to the softer exploratory forward pass? Is there an ablation that can separate these effects?

The deployed network remains a highly efficient Boolean circuit, but CovJac training requires continuous arithmetic operations such as codebook-distance computation, softmax, weighted averaging, and Jacobian operations. Is the efficiency motivation of the paper intended primarily for inference/deployment, or do the authors also expect an advantage for training hardware? If the latter, can the authors clarify what hardware architecture they envision for efficient CovJac training and whether any measured runtime or hardware results support this?

5 comments
AuthorsOpenReview ↗

(Part 1/2)

We thank the reviewer for the constructive feedback and for acknowledging the strengths: "a clear analysis of optimization difficulties" and "a coherent training approach" (Strength 1), an evaluation that is "relatively extensive", whose depth results "provide useful evidence" (Strength 2), and, in the summary, that the "deployed models remain strict Boolean gate networks". The main concern is significance relative to prior work: our contribution is a new training principle, and Proposition 7 shows that no affine product change of basis can replace it (see [W1]). On the training trade-off, the efficiency motivation of logic gate networks is deployment; our method focuses on training an accurate model to deploy, and its training is lighter than Soft-Mix and needs no special initialization (see [W2, Q2]). The ablation the reviewer asks for is now in the revised PDF (see [Q1]). Line, table, and appendix numbers refer to the submitted PDF unless marked "revised". In the revised PDF, revised and added text is shown in color; new references are listed in the bibliography.

[W1] What does the step from prior work add? It adds a new training principle, not a refinement of the representation. Recent methods learn a relaxed probability over the gates or over truth-table entries. We instead fit each gate's four coefficients by their distance to the 16 Boolean gates (Eq. 4) and train them through the exact gradient of that fit, the covariance gradient (Eq. 5). The first fix one would try is a change of basis, but Proposition 7 proves that no affine product basis, Walsh included, removes the straight-through starvation of the interaction coefficient without losing coherence or adding bias, while the covariance gradient reaches that coefficient on every sample (Proposition 5). The paper states this contribution from the start: it is not the representation, which is textbook, "but how to train in it" (L80). Our replies to NCTb [related work] and Sght [W2] relate this to recent work.

The experiments follow this analysis. Four coefficients match 16-parameter Soft-Mix across seven datasets (Table 1). Training holds to L=12 at the default initialization, where Soft-Mix loses 37pp on CIFAR-10 (Table 20). With the forward held fixed, the covariance gradient accounts for most of CovJac's gain over straight-through training on the interaction-dense CIFAR-10 (see [Q1]).

[W2, Q2] Is the efficiency motivation for training or deployment? The efficiency motivation of logic gate networks is deployment: the trained model runs as one hard gate per neuron. Training an accurate model is the precondition for that, and it is our focus. Training is continuous in every DLGN method; we claim no training-hardware advantage or measured training speedup. Our training is, however, lighter than 16-way Soft-Mix and needs no special initialization, which matters when training resources are limited: it learns 4 instead of 16 parameters per neuron, needs a quarter of the optimizer memory, uses no more per-sample compute (App C), and is stable to L=12 at the default initialization, where LightDLGN and WARP-LNN report needing residual initialization at depth (revised Sec 5.2). The revised Sec 3 states this motivation and these training properties.

AuthorsOpenReview ↗

(Part 2/2)

[Q1] Can the covariance be isolated from the softer forward? Yes; the revised PDF adds this ablation (new, revised Table 2). In CovJac, the soft forward is how a neuron computes its output during training: a mix of the 16 gates, each weighted by how close it is to the four coefficients (Eq. 4), where Multilinear-STE uses only the nearest gate. The covariance gradient is the exact derivative of that mix with respect to the coefficients (Eq. 5); its off-diagonal links the interaction coefficient to the always-active constant term, so that coefficient receives gradient on every sample, whereas a straight-through backward reaches it only when a=b=1. Because the two form one method, the only way to isolate the covariance is to keep the soft forward and replace its gradient with a straight-through one, a hybrid used only for this ablation (the "soft forward + straight-through" column of Table R1).

Table R1. Forward-fixed ablation (new, revised Table 2) beside Multilinear-STE (Table 1); last-10 accuracy, mean ±\pm std, 3 seeds; gain = covariance minus straight-through backward. MONK's-2 has no Table 1 cell here (50k vs 10k iterations).

DatasetMultilinear-STE (Table 1)soft forward + straight-throughsoft forward + covariancegain (pp)
MNIST98.09 ±\pm 0.0298.20 ±\pm 0.0698.31 ±\pm 0.03+0.11
SVHN66.69 ±\pm 0.1167.77 ±\pm 0.1768.91 ±\pm 0.16+1.14
CIFAR-1056.02 ±\pm 0.2356.47 ±\pm 0.1758.95 ±\pm 0.13+2.48
MONK's-2-79.06 ±\pm 1.5187.60 ±\pm 1.71+8.54

Within the two soft-forward columns only the backward changes, so the gain in the last column is due to the covariance gradient, and it grows with interaction density, as Sec 5.3 predicts. CovJac's gain over Multilinear-STE thus splits into a forward part and a gradient (covariance) part: on CIFAR-10, +0.45pp comes from the softer forward and +2.48pp from the covariance gradient (of +2.93pp in total), and on SVHN and MNIST about half comes from each (+1.08 and +1.14 of +2.22; +0.11 and +0.11 of +0.22). On MNIST, SVHN, and CIFAR-10, the covariance arm reproduces Table 1 within noise.

Thank you for the detailed response. The new ablation addresses my main experimental concern and provides better evidence that the covariance gradient contributes to the improvement. The clarification regarding deployment efficiency also addresses my second question.

My main remaining concern is the overall significance of the contribution relative to recent related work. Please specify it more expliciately.

Reviewer_NCTbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper formulates differentiable logic gate networks training as a vector quantization problem. It also uses the known multilinear polynomial decomposition and shows input gradients vanish at uniform initialization for soft-mix. Additionally, it shows that no affine product basis fixes the fact that the straight-through training provides the interaction coefficient gradient on only 1/4 of samples. The proposed method (Multilinear-CovJac) trains the four coefficients through forward pass on a soft vector-quantization and uses codebook covariance to couple the interaction coefficient to the always-active constant channel, showing that using four times fewer parameters per neuron, it does not perform worse than soft-mix and at 12 layers it outperforms it sharply.

Critical strengths

Proposition 7 seems novel in this context even though narrow by construction (only affine product, no corner basis): claims that using STE prevents the interaction coefficient from receiving the gradients of every sample, and that no affine product basis can compensate this issue without introducing bias. Previous work studies the choice of the straight-through estimator and the sparsity of their codebook gradient in a different context. Interpretation in Proposition 5 is interesting: the off-diagonals couple c_ab to the constant channel, so c_ab receives gradients at every sample. The negative results on making tau learnable is interesting.

Critical weaknesses

Technical issues: the overall issue is limited novelty and misattribution of previous contributions. Details:

  • Some works on representation + discretization are already published but not cited, e.g., LLNN, WARP-LUT, WARP Logic Neural Networks (which also release code). So the contribution "how to train in a four-dimensional coefficient space" is not entirely new. WARP-LNN proves the 2^n Fourier coefficients are the minimal exact parametrization, which entails the observation in §3. Another uncited relevant work is Polynomial Surrogate Training, whose binary case is the same as Multilinear-STE.

  • CovJac is basically as-is from Agustsson et al. 2017, acknowledged in E.8 but not in contribution list.

  • Proposition 2 is rather a corollary of Rüttgers 2025 than a sharpening. The depth-collapse phenomenon is already described by Petersen in NeurIPS2024 and Rüttgers in NeurIPS2026.

  • Soft-Mix is run with Gaussian logit initialization only but Residual Initialization (Petersen et al. 2024) fixes the depth collapse issue already. Deep Stochastic LGN (Kim, IEEE Access 2023) also use Gumbel / stochastic DLGN but is also uncited.

  • The paper's reproduction of IWP achieves a far lower performance than what's reported by Rüttgers, which may point to a reimplementation issue.

  • The authors do not release source code, therefore the results are not reproducible. The provided experimental results are insufficient to fully ensure that the repeated experiments match what is claimed in this paper

  • "STE underperforms even Soft-Mix, confirming that c_ab starvation is the binding constraint" but does not provide data evidence or even an explanation on the mechanism that causes this.

  • Results do not compare with more traditional train-harder DLGN methods, e.g., to complement experiments in Figure 3, and do not show how changing various parameters of the proposed method impact the selected metrics. The metrics on y axes in Figures 3b and 3c should be more formally defined and explained why they are interested to prove the authors' thesis. For example, §5.4 comments data on plot 3b, but does not suggest a deep interpretation. The parameter on x axis on Figure 3d should be also formally defined and its usefulness/role explained.

Non-technical issues: the overall issue is manuscript clarity. Details:

  • The overall writing style is at times confusing, redundant, informal, disorganized, and ambiguous, making the reading experience unnecessarily painful. Presentation clarity and organization is one of the biggest shortcomings of the article itself, which is not a minor problem. A subset of illustrative examples hereafter. The research question is not clear from the introduction, i.e., before the a paragraph "Solution: ..." the problem is not clear. Figure 1 introduces undefined symbols and notations, and has a redundant caption that should go in main text. Furthermore, it's not clear what the blocks represent or the role of the arrows, et cetera. It is the only figure in the paper and these visual aspects should be clarified. The figure is generated by AI, which is not necessarily a limitation by itself, but in this case hampered clarity. Overall, the paper description of method and motivation requires quite some guesswork by the reader. At several points it's unclear which results are claimed to be novel, introduced by this work, and what's taken from the state of the art. Relevant related works are presented, but the authors in some instances do not explicitly mention what are the relevant limitation of those works and how they bridge to their work. An example: there's a long paragraph in §2.1 about LUT, but it's unclear how, explicitly, it relates to the proposed method. Another mistake is the inclusion of experimental results already at methodological explanation ("comparison of the two mechanism" paragraph), should be moved to the results analysis section. Please convert the 3D plot in Figure 2c into a 2D plot. In §5.2 Ablation, text refers to Table 22, but the main body should be self-contained without forcing the reader to look at appendix to be convinced of the base statements in main paper body. Appendix is for complementary results that would strengthen what is already clear from main body only.
Questions for authors

Just a small specific question: what makes soft-mix collapse exactly between 9 and 12? where does this collapse happen exactly? and would there be a similar collapse for CovJac?

I would reconsider my evaluation only if the paper were rewritten basically from scratch without such heavy AI usage and in a more organized, unambiguous, and understandable prose, introducing concepts formally before using and describing them with less words and more symbols, and clarifying what's new and what's reused. The paper's main message should be self-contained in the main body without forcing the reader to rely on the content in the Appendix. For example, IEEE papers are often easier to read. Reducing redundancy would allow reducing the overall paper length. All artifacts (code, trained models, etc) should be released at submission time, together with a reproducibility document that allows an independent researcher to easily reproduce the results identically.

2 comments
AuthorsOpenReview ↗

(Part 1/2)

We thank the reviewer for the pointers to recent work and for acknowledging the strengths: "Proposition 7 seems novel in this context", the "Interpretation in Proposition 5 is interesting", and the "negative results on making tau learnable". The summary also notes that, with "four times fewer parameters per neuron", the method "does not perform worse than soft-mix and at 12 layers it outperforms it sharply". We answer the main concern, "limited novelty and misattribution of previous contributions", first, in [related work]: what is new is the training principle. The other points follow in the order of the review: residual initialization (with the Question), the corner-basis row, code, the binding-constraint evidence, training baselines, and presentation. Line, table, and appendix numbers refer to the submitted PDF unless marked "revised". In the revised PDF, revised and added text is shown in color; new references are listed in the bibliography.

[related work] What is new relative to the recent work the review cites? What is new is the training principle and its analysis. We fit four multilinear coefficients to the Boolean gates by distance (Eq. 4) and train them through the exact gradient of that step, the covariance gradient (Eq. 5). We also prove why this avoids two failures that other approaches leave in place: the gradient cancellation of a 16-way probability over gates, and the starvation of the interaction coefficient under straight-through training, which no affine product change of basis removes (Lemma 3, Propositions 5 and 7). The recent methods learn a relaxed probability, over the 16 gates or over each truth-table entry (LightDLGN (IWP), WARP-LNN), so they do not use this principle; revised Table 5 (App B.6) compares the two element by element.

On attribution, the submission credits the representation as textbook (L80), the complementation pairing behind the cancellation to Ruttgers' Eq. 7 (L259-261), and the soft-VQ step to Agustsson 2017 (L203). The revised PDF states Proposition 2 as a restatement of Ruttgers' observation rather than a sharpening of it. It also adds that the minimal Walsh form is WARP-LNN's, that the depth collapse is known from Petersen 2024 and Ruttgers 2026, and that PST's binary case shares our representation, whose straight-through training the paper uses only as an ablation anchor (L383-384); PST links no code. LLNN's public code reaches 97.00 ±\pm 0.13 (3 seeds) at its native setup (new, revised Sec 2). What is ours is the soft-VQ step on the fixed Boolean codebook and the analysis of its gradient, whose off-diagonal couples the interaction coefficient to the always-active constant channel (Proposition 5). We thank the reviewer for pointing to WARP-LNN, WARP-LUTs, LLNN, PST, and Deep Stochastic LGN. WARP-LNN and PST appeared on arXiv in February 2026, in the months before our submission, and LLNN and WARP-LUTs in 2025; the revised PDF cites them all.

The first failure is the gradient cancellation. The 16 gates form 8 complementary pairs whose input coefficients are equal in magnitude and opposite in sign (Lemma 1). A free probability over the 16 gates, as in Soft-Mix, therefore cancels its own input gradient at uniform (Proposition 2) and shrinks it near uniform (L273-274), where its rank-4 update keeps it (L301-302). Weights set by the distance from four fitted coefficients can never be uniform (Lemma 3); the measured per-layer signal at initialization is about 0.9, versus 0.29 for Soft-Mix (Sec 5.4). The second failure is the starvation of the interaction coefficient under straight-through training, which a change of basis does not remove: Proposition 7 proves that no affine product basis, Walsh included, removes it without losing coherence or adding bias, while the covariance gradient reaches that coefficient on every sample (Proposition 5). We claim this principle and its analysis, not higher accuracy than these methods (see also our replies to Sght [W2] and 9qrh [W1]).

AuthorsOpenReview ↗

(Part 2/2)

[residual init, Q] Does residual initialization already fix the collapse, and where does Soft-Mix collapse? Residual initialization corrects the collapse post hoc; our method removes its cause and needs no special initialization. Residual initialization biases each gate toward pass-through; LightDLGN pairs it with its parametrization, and LightDLGN and WARP-LNN report needing such an initialization at depth (LightDLGN Sec 3.3 and 5.2; WARP-LNN, gradient-stability appendix). Our coefficients induce soft-VQ weights that can never be uniform (Lemma 3), so the cancellation point is unreachable from any initialization. The depth sweeps (revised Sec 5.2) show training stable at the default initialization, Gaussian with std 1. Our depth analysis is scoped accordingly: Observation 1 assumes near-uniform initialization of the gate logits (L819), the regime that residual initialization avoids.

On the Question, Soft-Mix collapses between L=9 and L=12 on CIFAR-10, falling from 54.76 to 20.79 (App E.10, Table 20). Observation 1 (informal) bounds the per-layer signal at initialization by 0.72 (L273-274), so 0.72^L is about 0.05 at L=9 and 0.019 at L=12. Because layers commit from the output toward the input, at enough depth the early layers do not commit within the training budget (L816-817), which is what the drop from L=9 to L=12 shows. CovJac does not collapse: it moves from 58.41 to 58.44 (Table 20).

[corner basis] Why is our corner-basis row below LightDLGN's reported result? The row is below because it is not LightDLGN's method. The row (77.95), labeled "IWP" in the submitted Table 22 and relabeled "Corner basis" in the revised Table 24, is the corner basis "on its own" (L1950), trained in our harness without their code (the linked repository is unavailable, as the WARP-LNN authors note) and without residual initialization: the run's recorded configuration has it off, as the revised App F.7 and Table 24 footnote now state. The row is therefore not comparable to their 94.02 on MNIST (their Table 1). The basis comparison is the straight-through row beside it (revised Table 24), where the canonical basis is 17.25pp above the corner basis.

[code] Can the results be reproduced? Yes, the revised PDF gives everything needed to reproduce the results: architecture, equations, all hyperparameters, wiring, and the 3-seed protocol. Source code will be released on acceptance.

[binding constraint] Is the binding-constraint claim supported? Yes: the paper gives the mechanism, and the revised PDF adds the data the reviewer asks for. The mechanism is that, under straight-through training (Multilinear-STE, the "STE" of the quoted claim), the interaction coefficient receives gradient only when a=b=1, which is 1 in 4 samples under balanced inputs (Sec 3.1, Lemma 6). App E.7 measures its gradient norm at 0.227 of the constant coefficient's, close to 1/4, and Sec 5.4 measures 0.42 on CIFAR-10, whose thresholded inputs make a=b=1 more frequent. The data show that, with the soft-VQ forward held fixed, the starved arm stays near the straight-through level on CIFAR-10 (56.47 in revised Table 2, vs 56.02 in Table 1), while the covariance gradient recovers +2.48pp on CIFAR-10 and +8.54pp on MONK's-2 (revised Table 2; our reply to 9qrh [Q1]). The quoted claim concerns CIFAR-10/100, where straight-through training trails Soft-Mix (Table 1); on CIFAR-10, the ablation traces this to the starvation, not to the forward.

[train-harder] Are stronger training baselines and sensitivity covered? Yes: longer training, convergence speed, and the one added hyperparameter are below; WARP-LNN is compared in our reply to Sght [W2], and the revised Sec 2 relates the recent training methods. At depth, Soft-Mix converges slowly, the slowdown the cancellation predicts (Observation 1): doubling the iterations raises it to 92.08% at L=12 on MNIST, still 6.2pp below CovJac, which reaches 98.30% at half that budget (revised Table 20). CovJac also converges fastest: on MNIST it first reaches 98% at 2k to 3k iterations, against 5k to 6k for Soft-Mix and 16k to 23k for Gumbel-ST (new, from the Table 1 training logs). The only hyperparameter CovJac adds is the soft-VQ temperature tau, set to 1 on every dataset except Adult (0.1); on MNIST, accuracy stays within 0.22pp across tau in [0.3, 5.0] (revised App E.13).

[presentation] Is the presentation clear? The revised PDF addresses several of the examples the review gives: it states the research question in the introduction, adds a legend to Figure 1, and defines the Figure 3 metrics in its caption. It also moves the train-deploy measurement from the method section to Sec 5 and ties the related work, including the Sec 2.1 LUT paragraph, to the method (see also our reply to Sght [W3]).

Comments on the paper

1 comment

We thank the reviewers for their feedback and the Area Chair for handling this submission. Every point raised is answered, the main ones below and the rest in the individual replies. The paper helps explain how logic gate networks learn to select each gate from a fixed library, the 16 Boolean gates, and it points to scaling them to larger systems without post hoc or ad hoc tuning. Below, section, table, and appendix numbers refer to the submitted PDF unless marked "revised".

Significance relative to related work. Our contribution is distance-based selection with its exact gradient, and an analysis of how the training gradient reaches the four coefficients of a gate in the multilinear polynomial basis. Recent methods (LightDLGN, WARP-LNN, PST) also learn 4 parameters per gate. Ours differs in how the gate is chosen and trained: by distance to the 16 Boolean gates, through the exact gradient of that selection. The analysis gives two criteria to check a training method before it is run. Its weights must not reach the point at which a softmax over the 16 gates cancels its own input gradient, and its gradient must reach the interaction coefficient on every sample. Our selection meets both (Lemma 3, Proposition 5), whereas straight-through training gives the interaction coefficient gradient on only a quarter of the samples (Lemma 6), and no affine product basis removes this starvation (Proposition 7). The coupling also changes which gates are selected: our method selects almost no constant gates (<0.5%, against 4-10% for Soft-Mix, Gumbel-ST, and Multilinear STE), so almost every neuron computes a function of its inputs (Sec 5.4, Figure 5). We added a contribution table that separates what is ours from prior work (revised Table 5, revised App B.6).

Strengths the reviewers noted. NCTb notes that Proposition 7 "seems novel in this context" and finds the Proposition 5 interpretation "interesting". 9qrh finds "a clear analysis of optimization difficulties" and that "the depth results provide useful evidence that the proposed training approach can improve stability while preserving a Boolean circuit at deployment". Sght calls the problem "important", the degeneracy insight "interesting and relevant", and the method one that "makes sense".

Main concerns.

  1. Novelty relative to prior and recent work. What is ours is stated above. WARP-LNN, PST, and WARP-LUTs, three of the recent works the reviewers pointed to, are arXiv preprints (we had not seen them when writing). The revised PDF cites all three, together with LLNN and Deep Stochastic LGN. Details are in the replies to NCTb [related work], 9qrh [W1], and Sght [W1] and [W2].

  2. Whether the gain comes from the covariance gradient or from the softer forward pass (9qrh [Q1]), and whether the starvation of the interaction coefficient is the binding constraint (NCTb [binding constraint]). A new ablation answers both; it is listed below.

  3. Comparison with recent training methods, at least LightDLGN and CAGE (Sght [W2]). Where public code exists, we ran it; the WARP LNN run is listed below. CAGE (Kim, ICML 2026) keeps the 16-way gate and adapts its backward temperature, a different axis from ours; at temperature 1 it reaches accuracy similar to Gumbel-ST (its Fig 2b). The linked repository of LightDLGN is unavailable, so we compare its corner basis with ours in one harness. Details are in the replies to Sght [W2] and NCTb [corner basis].

  4. Other points, each answered in the individual replies: training cost (9qrh [W2, Q2]), residual initialization and depth (NCTb [residual init, Q]), train-harder baselines and parameter sensitivity (NCTb [train-harder]), reproducibility (NCTb [code]), presentation (NCTb [presentation]; Sght [W3]), and the Adult dataset (Sght [ethics]).

Added in the rebuttal.

  • An ablation that keeps the soft-VQ forward pass and replaces only the covariance gradient with a straight-through one. The covariance gradient adds +0.11pp on MNIST, +1.14pp on SVHN, +2.48pp on CIFAR-10, and +8.54pp on MONK's-2 (3 seeds each), growing with the interaction demand of the task (revised Sec 5, revised Table 2).
  • WARP-LNN, run from the public torchlogix implementation in our MNIST harness, reaches 98.09 ±\pm 0.02 (3 seeds), comparable to the methods in our tables (reply to Sght [W2]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.