- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The submission studies the training of logic gate networks (LGNs). It makes some technical observations, in particular that the originally proposed 16-gate softmax formulation has degenerate gradients, and it proposes two new techniques based on quantization to the nearest codebook entry, which require fewer parameters per gate than the softmax (though not fewer than related works). Experiments in typical LGN benchmarks show improvements over classic baselines, but a apples-to-apples comparison to recent alternative training techniques is missing.
Critical strengths
-
The manuscript addresses an important problem in the context of logic-gate networks: improving their training.
-
The insight of degeneracy in the optimization are interesting and relevant.
-
The proposed method makes sense and seem to lead to improvements.
Critical weaknesses
-
The technical depth and breadth of the result are limited. The problems of training in the 16-gate SoftMix representation have been reported before, even though less formally, and most recent LGN works do not use plain SoftMix for training anymore. While the reported insights follow from rather elementary observations and computations. While they are good to know, they might be better batch to topical workshop. The proposed new methods are ad-hoc suggestion, that make sense, but do not stand out compared to methods suggested previously in the literature.
-
The submission lacks comparison to important related work. Conceptually, a comparison to CAGE (Kim, ICML 2026) would be important, as that starts from the same observations and also proposes an alternative way of training. Experimentally, the comparison just to plain baselines (SoftMix, Gumbel-ST) is insufficient, and other recently proposals should be included, at least LightDLGN (Ruttgers etal, ICLR 2026) and CAGE.
-
The writing is not clear enough for publication. It relies too much on technical terms and implicit assumptions, while also arguing with concept that are not truly defined (e.g. "interaction coefficient starvation").
Questions for authors
I do not have questions that could materially affect my assessment.
3 comments
(Part 2/2)
[ethics] Is the Adult dataset appropriate? We use Adult only for comparability with prior DLGN work, not to study fairness. The revised Ethics statement says so and cites Retiring Adult (Ding et al. 2021) for its limitations. We also ran ACS Income, the replacement that Retiring Adult proposes (new, revised App E.14; 3 seeds): CovJac reaches 81.95 0.03, against 81.85 0.05 for Soft-Mix, 81.81 0.02 for Multilinear-STE, and 81.76 0.05 for Gumbel-ST, so the four methods tie within 0.2pp, as they do on Adult (within 0.3pp).
Thank you for your response. I have no questions at this point and will maintain my score while awaiting the upcoming discussion with the other reviewers and the area chair.
(Part 1/2)
We thank the reviewer for the detailed comments and for acknowledging the strengths: "an important problem" (Strength 1), insights that "are interesting and relevant" (Strength 2), and a method that "makes sense" (Strength 3). On the missing "comparison to recent alternative training techniques": the revised PDF cites the recent methods, which differ from ours in kind, and WARP-LNN, run in our harness, is comparable to the methods in our tables (see [W2]). Our 4-vs-16 reduction is relative to the 16-gate softmax, not a claim against these works. On technical depth, Proposition 7 is an impossibility result over a design space, not an elementary observation (see [W1]); on clarity, see [W3]. Line, table, and appendix numbers refer to the submitted PDF unless marked "revised". In the revised PDF, revised and added text is shown in color; new references are listed in the bibliography.
[W2] How does this work differ from CAGE, LightDLGN (IWP), and other recent methods? Our method differs from these in kind, not only in basis. Soft-Mix, Gumbel-ST, and architecture-search relaxations learn a probability over the 16 gates. CAGE (Kim, ICML 2026) keeps that 16-way gate and adapts its backward temperature for robustness; in its accuracy-versus-temperature figure (Fig 2b), it differs from Gumbel-ST only at low temperature. At temperature 1, the Gumbel temperature of our Gumbel-ST baseline, the two reach similar accuracy (about 98% on MNIST), so our tables include Gumbel-ST as the closest stand-in. LightDLGN and WARP-LNN learn, through an output activation, the probability that each truth-table entry is 1, and use residual initialization at depth. We instead fit the gate: four multilinear coefficients, with the gate selected by its distance to the 16 Boolean gates, softly in training (soft vector quantization, Eq. 4) and as the nearest gate at deployment (Eq. 2), with no output activation or added loss. The difference matters. Because complementary gates have opposite input coefficients (Lemma 1), a free probability over the 16 gates, as in Soft-Mix, cancels its own input gradient at uniform (Proposition 2) and shrinks it near uniform. Weights set by distance from four fitted coefficients are never uniform (Lemma 3), and the exact gradient of the fit reaches the interaction coefficient on every sample (Proposition 5). We claim this principle and its analysis, not higher accuracy than these methods (see also our reply to NCTb [related work]). We thank the reviewer for pointing to CAGE and LightDLGN; the further recent methods are discussed in our reply to NCTb [related work].
Where public code exists, we ran it; for LightDLGN, whose code is unavailable, we compare the bases. WARP-LNN, run from the public torchlogix implementation in our MNIST harness (new; L=6, same width, batch, and learning rate; 3 seeds), reaches 98.09 ± 0.02 with residual initialization and 98.03 ± 0.01 with random initialization, comparable to the methods in our tables. On MNIST, CovJac first reaches 98% at 2k to 3k iterations (Table 1 runs), and WARP-LNN at 18k to 23k (these runs). LightDLGN's linked repository is unavailable, so we compare bases in one harness under straight-through training: the corner basis reaches 80.84, against 98.09 for Multilinear-STE, our canonical basis (Table 22; details in our reply to NCTb [corner basis]). These numbers describe the bases under our training, not LightDLGN's method, which trains differently.
[W1] Are the results elementary? They are not: Proposition 7 is an impossibility result over a whole design space, showing that no affine product basis attains full coverage, gradient coherence, and zero straight-through bias at once (proved in App A.6). Together with Proposition 5 and Lemma 3, it predicts, before any experiment, which methods starve and which collapse at depth, and the experiments follow that prediction. The earlier, less formal reports of the Soft-Mix problem are credited (Sec 2), and none contains this impossibility result. Nor is plain Soft-Mix our only baseline: Gumbel-ST, the stochastic hard selection used in recent DLGN training (Kim 2023; Yousefi et al. 2025), is in every main and depth table.
The method follows from the same analysis and is not ad hoc. There is one proposal, Multilinear-CovJac, trained by the exact gradient of the soft-VQ forward (Eq. 5), with Multilinear-STE as the ablation anchor (L383-384); [W2] describes how it differs from recent methods.
[W3] Are the terms defined? The term the reviewer names stands for one statement, made in Sec 3.1: under straight-through training, the interaction coefficient "receives gradient on only 1 in 4 samples" (under balanced inputs, Lemma 6). Its earlier uses, in the abstract and the contribution list, carry the same meaning. Our reply to NCTb [presentation] lists the clarity changes.