acceptodds
Under review as a conference paper at ICLR 2027

Logic-Native Transformer: Mapping Transformer Computations to Trainable Logic Gate Networks

Abstract

The growing demand for deploying Transformers on resource-constrained edge devices motivates the design of hardware-efficient neural architectures. Differentiable logic gate networks offer one such approach by directly learning compositions of the Boolean operations underlying digital hardware, enabling representation learning in feedforward, convolutional, and recurrent architectures. Extending this approach to Transformers requires computing attention with Boolean operations and overcoming the optimization difficulties of deeply stacked logic gates. To address these challenges, we propose a Boolean attention mechanism based on XNOR similarity, top-k selection, and majority vote, and introduce XOR residual connections. Integrating these components with learnable logic gate modules yields a scalable logic gate network vision transformer (LGN-ViT) with a unified Boolean inference architecture. On CIFAR-10, LGN-ViT reaches a depth of 96 learnable logic gate layers, 6.4 times that of the convolutional logic gate baseline. For width scaling, our analysis shows 26.8% higher retention of marginal accuracy gains for LGN-ViT than for this baseline under a 3.35-fold increase in trainable parameter count.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Differentiable Logic Gate Networks, Transformers, Boolean Attention, Edge Intelligence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_smaUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LGN-ViT, a Vision Transformer implemented using trainable logic gate networks. Motivated by efficient inference on resource-constrained edge devices, it combines XNOR-based similarity, top-k selection, majority voting, and XOR residual connections with discrete forward training and differentiable backward surrogates. Experiments report accuracies of 57.75% on CIFAR-10 and 98.11% on MNIST.

Critical strengths
  1. The paper presents a coherent logic-native Transformer architecture. It integrates learnable feature transformations, content-dependent token interactions, and binary residual updates into a unified Boolean inference framework.
  2. The training components are supported by targeted ablation experiments.
Critical weaknesses
  1. The experimental results are not convincing enough. In Table 1, ConvLogic S achieves 60.38% accuracy on CIFAR-10 with 1.337M parameters and a 160.3K estimated logic budget, whereas LGN-ViT D768 only achieves 52.56% with 1.478M parameters and a 700.4K budget.
  2. The evaluation is limited in scale and task diversity. The proposed LGN-ViT is evaluated only on CIFAR-10 and MNIST.
Questions for authors
  1. Could the author(s) evaluate LGN-ViT on more challenging benchmark?
  2. Could the author(s) report end-to-end inference latency and memory usage on a target hardware platform?
3 comments
AuthorsOpenReview ↗

Response to Reviewer smaU: Accuracy, Evaluation Scope, and Hardware Performance

We thank you for your careful review and for recognizing the coherence of LGN-ViT's unified Boolean inference framework and the evidence from our targeted ablations. Your feedback has helped us strengthen the experiments and clarify the relationship between model performance and resource requirements.

1. Classification accuracy and model efficiency

To address the accuracy gap you identified, we refined how binary features are used within the LGN-ViT framework. LGN-ViT-SR achieves 64.42% CIFAR-10 test accuracy with 1.330M parameters, exceeding ConvLogic S's 60.38% with 1.337M. Compared with the original D768 model you highlighted, this is an improvement of 11.86 percentage points, with approximately 10% fewer parameters. These results further support the effectiveness of the core LGN-ViT architecture. The methods and full results are provided in our general comment and Appendix B.4, Table 13 (p. 26) of the revised PDF.

ConvLogic remains more accurate at comparable logic budgets. This comparison should be understood in the context of how the architectures organize computation: convolutional models directly exploit locality in images, while LGN-ViT's logic budget also includes matching, selection, and aggregation circuitry for input-dependent interactions. These operations establish token connections based on the current input, and their resource costs and accuracy benefits need to be evaluated for the task at hand.

Comparisons of floating-point models provide related context. ViT (ICLR 2021) explains that CNNs' built-in locality and translation equivariance aid generalization with limited data, while ViT relies more on learning spatial relationships from data. In CIFAR-10 transfer experiments with ImageNet pretraining, Touvron et al. (ICML 2021), Tables 5-6 report 97.9% with 307M parameters for ViT-L/16, versus 98.9% with 66M for EfficientNet-B7. Adopting a Transformer therefore does not automatically guarantee better parameter efficiency. Our new results show that refinements suited to Boolean features can substantially improve the accuracy-parameter tradeoff within LGN-ViT.

2. Evaluation scope and more challenging benchmarks

We agree that evaluation on more challenging vision and sequence benchmarks would further test the value of input-dependent attention. We chose MNIST and CIFAR-10 to use common evaluation benchmarks with ConvLogic. The paper also examines width scaling, depth scaling, and component ablations to assess the trainability of a logic-native Transformer and the roles of its key designs.

Our work establishes the computational basis for building and training Transformers with differentiable logic gate networks, providing an independent architectural backbone for future adaptations to different tasks. The new LGN-ViT-SR results further demonstrate practical gains from improving feature utilization within this framework. Building on this foundation, extending the model to more complex vision and sequence tasks is an important direction for our future research.

3. End-to-end inference latency and memory usage

For CIFAR-10 D192 (5 blocks, 65 tokens, three 64-bit heads, top-k=15), we implemented a clocked RTL selection-and-aggregation engine with full datapath and control for register state, active-mask updates, gather/vote accumulation, and block/query scheduling. We verified its outputs against the software reference in VCS and synthesized it using a 28 nm standard-cell library (see Appendix B.3 and Table 12, p. 22 of the revised PDF).

With score and value inputs ready, VCS simulations report 45 cycles per query/head for a single engine. Three head-local engines operate in parallel, processing all three heads in the same 45 cycles with a total synthesized cell area 2.98 times that of a single engine. Each engine uses 64 comparison nodes and is reused across queries and blocks to reduce hardware replication. With continuous score and value streaming and a 46-cycle query initiation interval, the three-engine unit completes selection and aggregation across all five blocks (65 queries per block, 325 per head) in 14,949 cycles. At the 3 ns target clock period, this corresponds to 44.847 μs.

Bit-packed binary Q/K/V payloads occupy 4,680 bytes per block. Two 65-entry, 7-bit score-row buffers per head require 2,730 bits across all three heads. This supports row-wise score streaming without storing the full attention-score matrix. These results establish functional equivalence, deterministic timing across blocks, and compact on-chip buffering for our core attention architecture.

Thank you again for your careful assessment and constructive suggestions.

Thank you for the response. My score remains unchanged.

AuthorsOpenReview ↗

Thanks For Recommending Acceptance

Thank you very much for your positive assessment and recommendation for acceptance! We really appreciate your time.

Reviewer_q2yJOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Logic Gate Network XNOR-based attention mechanism with top-k selection and majority voting to transfer the concept of self-attention to a trainable LGN. The paper also proposes to include residual connections to improve scalability and training stability. The attention mechanism is used to implement a LGN-ViT architecture. The model is evaluated on CIFAR-10 and MNIST. The model performs roughly on par with standard LGNs and seems to perform a bit lower than convolutional LGNs. The paper provides a solid ablation of all major components.

Critical strengths
  • Good, novel idea: The overall idea of the paper to frame attention base on XNOR gates is interesting and to my best knowledge not published so far.
  • The architecture seems solid.
  • The paper is well written and good to follow.
Critical weaknesses
  • Experimental results: My main concern is that the performance of the model can not keep up with the respective convolutional architectures. Compared to the standard LGN, the performance seems to scale equally, but it has not yet reached the point where it would show a clear benefit.
  • Small-scale experiments: Related to this problem is the fact that the experiments are run on a minimal dataset, most likely due to the high training costs. It is therefore possible that a performance increase on this architecture might only show with larger models and/or more data, but that this would not be feasible to train in an academic context.
Questions for authors

I'm really on the fence with this submission, so the rating from my side is more of a strong borderline. I really like and appreciate the idea, but based on the current performance, I have the feeling that the proof of whether this method will be useful or not will be left to others. I also understand that scaling up is not always possible. I would therefore wait for the feedback of my fellow reviewers as well as the rebuttal before making a final decision.

Post rebuttal comment: The new provided evaluation were very convicing and clearly show the advantage of the proposed method. I'm increasing my score from 2 to 4.

4 comments
AuthorsOpenReview ↗

Response to Reviewer q2yJ: Accuracy and Scaling

We sincerely appreciate your recognition of the novelty and coherent design of our logic-native Transformer, as well as your positive feedback on the clarity of the paper. We are particularly grateful for your understanding of the practical resource constraints on scaling experiments in an academic context.

Our additional experiments show that substantial accuracy gains are already achievable at modest parameter counts through targeted refinements within the LGN-ViT framework. LGN-ViT-SR reaches 64.42% accuracy with 1.330M parameters, compared with 60.38% and 1.337M for the published ConvLogic S. At the three smaller parameter scales, it also exceeds our custom ConvLogic configurations by 5.92–8.86 percentage points at comparable parameter counts. As detailed in our shared response and Appendix B.4 of the revised PDF, these results provide concrete evidence of the benefits achievable within the existing framework and further support the value of its core architecture.

Our work provides a proof of principle for constructing and training Transformers with differentiable logic gate networks. Widely used training frameworks such as PyTorch benefit from mature CUDA optimizations for standard floating-point operations. For Boolean-based models, efficient differentiable training involves substantial custom CUDA development because their computations differ from those of conventional real-valued networks. In LGN-ViT, input-dependent token matching, selection, and aggregation introduce additional optimization challenges compared with feed-forward and convolutional LGNs.

We have already invested substantial engineering effort in implementing and optimizing dedicated CUDA kernels. However, efficiently scaling experiments to larger models and datasets requires continued optimization of the full training pipeline within our available compute budget. Building on the feasibility established here, we will continue this implementation work to support more efficient, lower-cost evaluation at larger scales and across a broader range of tasks.

Thank you again for your constructive feedback and encouragement.

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer q2yJ,

We sincerely appreciate your recognition of the novelty of our idea and your positive assessment of our architecture.

We have provided detailed responses to all the concerns you raised in our response above and additional comments. If you have any remaining questions or concerns, we would be happy to provide further clarification.

If you feel that our responses have adequately addressed your concerns, we would be grateful if you could reconsider your assessment accordingly.

Thank you again for your time and consideration!

Best regards,
The Authors

Reviewer_q2yJOpenReview ↗

Re: Looking Forward to Your Further Feedback

Dear authors,

Sorry for the late reply. The additional evaluations are convincing. I'm happy to increase my score from 2 -> 4. I recommend incorporating the new evaluations into the camera-ready version of the paper. I wish the authors all the best for the remainder of the process.

AuthorsOpenReview ↗

Our Sincere Thanks for Your Response and Recognition

Dear Reviewer q2yJ,

Thank you very much for your timely response and recognition of our work. We are delighted that you found the additional evaluations convincing. We will certainly incorporate these additional evaluations and results into the camera-ready version if we are fortunate enough to have our paper accepted.

Thank you again for your time, thoughtful suggestions, and positive assessment.

Best regards,
The Authors

Reviewer_YU5MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the open problem of how to model the transformer architecture and the underlying attention mechanism as a differentiable logic gate network (DLGN). Prior work has so far only modelled FNNs, CNNs and RNNs as DLGNs. Designing a DLGN architecture that retains the information-theoretic capacity of attention but remains efficiently optimizable is hence an open, non-trivial and relevant problem.

Contributions: The paper proposes several novel and non-obvious design choices

Firstly, they propose a differentiable logic gate (DLG) parametrization of attention.

Secondly, they propose an alternative design of residual connections in DLGNs. While Petersen et al. (2024) used residual initializations and Rüttgers et al. (2026) explored residual connections as direct wire connections to subsequent layers, this work takes a different perspective and employs XOR residual connections. This novel perspective regards the output f(x)f(x) of Logic Multi-Head Attention (MHA) and FFN as a bit-wise delta to the original input xx, just as the original residual connection in standard architectures can be regarded as the addition of a delta.

Thirdly, in the final classification head, they make use of a convex interpolation from a Multi Layer Perceptron (MLP) towards a DLGN during training. Starting from α=1\alpha=1, the head interpolates f(x)=αfMLP(x)+(1−α)fDLGN(x)f(x)=\alpha f_{MLP}(x)+(1-\alpha)f_{DLGN}(x) towards α=0\alpha=0, where the full DLGN is recovered. They observe that the gradients that the MLP backpropagates to earlier layers provide a better supervision than the DLGN.

With these design choices, they build a DLGN optimization framework for the transformer architecture that seems scalable and sound.

Validation The authors instantiate these layers as a Vision Transformer and train it on MNIST and CIFAR-10. They scale it to dozens of layers and compare it to prior FNN and CNN architectures from Petersen et al. (2024). However, their architecture falls below these baselines in the performance-efficiency pareto frontier on both datasets, see also Table 7 in Appendix B.1.5 for a full comparison. The best DLGN Vision Transformer achieves 57% on CIFAR-10, while the CNN architecture from Petersen et al. (2024) with a matched logic gate budget achieves 80% accuracy. Therefore, their design proposals cannot advance that frontier further, and unfortunately remain only of ideological appeal for now.

Critical strengths

1. The paper addresses a relevant problem at the efficiency-frontier of large-scale models whose solution requires design exploration of architecture and optimization that is not straightforward. The authors explore this open design problem space in an reasonable way, as they build a lot upon established design choices and insights from prior work (logic gate trees, IWP, XNOR similarity, STE with surrogates). The proposed ideas are thus novel but often still feel intuitive.

The paper proposes several novel and non-obvious design choices: logic attention, XOR residual connections, MLP-DLGN interpolation (see details above).

Critical weaknesses

1. Their architectures fall below the baselines We do not build a transformer for their own sake, we build them because they are have proven more powerful in several long-sequence modelling tasks. However, the evaluations of this work show no advantages of the DLGN transformer. The FFNs and CNNs from Petersen et al. (2024) mostly achieve superior performance at matched logic gate budgets and fewer logic gates at matched performance levels. The ablation studies provide some insight which design choices are beneficial to the overall result, but they do not set a new state-of-the-art despite the information-theoretically more powerful architecture. Without positive results in validation, the audience can not faithfully adopt these design choices into their own research work.

2. More challenging baselines are missing Evaluations are only done on low-scale image classification benchmarks with 10 classes, MNIST and CIFAR-10. Previous work - the RNN architecture in Bührer et al. (2025), the IWP reparametrization in Rüttgers et al. (2026) - have already advanced to more challenging vision and language benchmarks: CIFAR-100, ImageNet32 and WMT'14. I would expect this work that designs an architecture that is information-theoretically more powerful in long-sequence modelling tasks to show superior performance on adequately challenging benchmarks.

Questions for authors
  • For what reason do the LGN ViTs lack behind LGN CNNs and FNNs? What prevents them from converging to more accurate approximations?
  • In their estimated token budget for their LGN MHA, the authors share "selection and aggregating circuitry across candidates, queries and encoder blocks". Is this a realistic assumption? There are O(T2)O(T^2) candidates in total per encoder block, and pipelining them through a single shared circuitry might not achieve sufficient throughput and latency.
  • For what reason have the authors chosen to hard-code a top-k selection rule and not a top-p selection rule into their attention design? If all kk selected embeddings are averaged via majority, then the absolute value of their XNOR similarity score does not affect downstream computation. However, these similarity score might be highly different for a fixed kk, especially towards convergence.
4 comments
AuthorsOpenReview ↗

Response to Reviewer YU5M: Accuracy, Hardware Sharing, and Attention Design

We sincerely appreciate your detailed and expert assessment of our Boolean attention, XOR residual connections, and training approach, and your recognition of their novelty and technical merit.

1. What explains the accuracy gap, and what do the new results show?

Our analysis found that the encoder learns useful binary spatial features that the classification readout does not fully exploit. To use these features better, we encode regional activation densities as binary inputs and organize the FFN as a key-value memory within LGN-ViT. LGN-ViT-SR achieves 64.42% on CIFAR-10 with 1.330M parameters. ConvLogic S achieves 60.38% with 1.337M. Full results appear in our general comment and Appendix B.4 of the revised PDF (Table 13, p. 26).

Under the paper's accounting, SR D228 achieves 64.42% at an estimated logic budget of 1,246.8K, compared with 62.14% and 12,288.0K for the most accurate DiffLogicNet configuration in Table 7 (p. 18). ConvLogic remains more accurate at comparable logic budgets. The new results substantially improve our previous accuracy-budget tradeoff.

2. Why MNIST and CIFAR-10?

We chose these datasets to align with ConvLogic's evaluation. We agree that more challenging vision and sequence benchmarks would further test the value of input-dependent attention. Convolutional and recurrent logic networks organize information flow through local connections or recurrent states (Petersen et al., 2024; Bührer et al., 2025). Unlike them, LGN-ViT introduces explicit query–key matching to select and aggregate tokens according to the current input, establishing an independent, logic-native Transformer backbone for future adaptations to different tasks.

3. Parameter counts and accuracy in ViT versus CNNs

LGN-ViT follows the encoder organization of ViT. The ViT paper explains how CNNs' locality and translation equivariance aid generalization with limited training data. In CIFAR-10 transfer experiments with ImageNet pretraining, Touvron et al. (2021), Tables 5-6 report 97.9% for ViT-L/16 with 307M parameters, versus 98.9% for the CNN EfficientNet-B7 with 66M. ViT therefore does not automatically outperform CNNs in parameter efficiency.

4. Is circuit sharing realistic, and how does it affect throughput and latency?

Our RTL implementation reuses comparison and aggregation circuitry across candidate selections. The same engines can serve successive queries and encoder blocks. The O(T2)O(T^2) query-key pairs determine the workload; physical resources depend on the number of parallel engines.

Once scores and values are available, one D192 engine produces one head output in 45 cycles; three parallel engines produce three head outputs in the same 45 cycles at 2.98 times the cell area. See Appendix B.3 (p. 21) and Table 12 (p. 22) of the revised PDF. This area-throughput tradeoff is consistent with FINN's folding approach. The timings cover selection and aggregation; end-to-end latency also depends on scoring, data supply, and scheduling.

5. Why Top-k rather than Top-p, and why not weight values by absolute similarity?

XNOR similarity satisfies sim(q,k)=Dh−dH(q,k)\mathrm{sim}(q,k)=D_h-d_H(q,k), where dHd_H is Hamming distance. Thus, Top-k retrieves the nearest keys in the learned binary space. For odd KK, bitwise majority satisfies

Maj⁡(v1,…,vK)=arg⁡min⁡b∈0,1Dh∑i=1KdH(b,vi).\operatorname{Maj}(v_1,\ldots,v_K)=\arg\min_{b\in\\{0,1\\}^{D_h}}\sum_{i=1}^{K}d_H(b,v_i).

Similarity therefore determines which values are retrieved, while majority voting returns the binary vector that minimizes their total Hamming distance. Fixing an odd KK gives a fixed aggregation workload and avoids tied votes. XNOR counts are not probabilities: Top-p requires a probability-mass mapping and an aggregation rule for variable candidate counts. Query-dependent Top-k selection also has precedent in Gupta et al. (2021).

6. Research significance of LGN-ViT

Our work provides a proof of principle for building and training a Transformer with differentiable logic gate networks and input-dependent Boolean attention. This extends trainable LGNs from feed-forward, convolutional, and recurrent architectures to Transformers. Together with the fast inference and compact logic implementations demonstrated by Petersen et al. (2024), this provides computational and training foundations for future logic-native NLP Transformers and Logic LLMs, which may open new paths to deploying large models on resource-constrained edge devices.

Thank you again for your detailed assessment and constructive feedback.

AuthorsOpenReview ↗

Looking Forward To Your Further Feedback

Dear Reviewer YU5M,

We sincerely appreciate your detailed feedback and careful review.

We have provided detailed responses to all your questions regarding accuracy and efficiency, benchmarks, circuit sharing, and attention design in our response above and additional comments. If you have any remaining questions or concerns, we would be happy to provide further clarification.

If you feel that our responses have adequately addressed your concerns, we would be grateful if you could reconsider your assessment accordingly.

Thank you again for your time and consideration!

Best regards,
The Authors

I thank the authors for the additional experimental results on larger-scale datasets, explanations and references. I think their contribution is now sufficiently complete to justify publication as a poster. While the overall performance of their DiffLogic ViT is still not competitive to standard architectures, their innovations constitute a first foundation that future work on how to learn logic-native sequence models can build upon.

To set a proper foundation for others, I strongly encourage the authors to open-source their code to stimulate progress on these open problems.

Below is a more detailed assessment of the author's replies:

Superior performance on larger-scale datasets

To overcome the performance gap to ConvLogic, the authors introduced an architectural extension that aggregates features across neighboring patches. That way, they incorporate a similar spatial inductive bias as is used in CNNs.

Their refined DiffLogic ViT now achieves superior performance to DiffLogic CNNs on a larger-scale dataset Tiny-ImageNet. This aligns with the claim in the original ViT paper that ViTs cannot outperform CNNs if they are not trained on sufficient data. To the best of my knowledge, they set a new state-of-the-art performance on larger-scale image datasets among DLGN architectures.

On top-k XNOR similarity selection

What I am concerned about is that the top-kk key vectors are averaged irrespective of their XNOR similarity value. If the top 1/2 XNOR similarity scores are far larger than the rest, irrelevant key vectors are effectively up-weighed in majority. There are several alternatives to this formulation that could be explored and compared against this selection mechanism.

But as their architecture now already exhibits superior performance, this can be left to future work.

Rating

I have hence increased my rating from 2 to 3 (weak accept).

AuthorsOpenReview ↗

Our Sincere Thanks for Your Recognition and Suggestions

Dear Reviewer YU5M,

Thank you very much for your recognition of our contribution, recommendation for acceptance, and valuable suggestions. We will follow your suggestion and conduct relevant experiments in future research.

We would like to share the code for this work. It needs internal sanitization before public release, and we will make it available once ready. We sincerely hope that our work will benefit the research community, and we warmly welcome feedback to help us improve it.

We will incorporate the additional results into the camera-ready version if our paper is accepted.

Best regards,
The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Additional CIFAR-10 Results at Comparable Parameter Counts

We thank the reviewers for recognizing the value of our Boolean attention and logic-native Transformer framework, and for highlighting the need for stronger accuracy results. To address the accuracy gap to ConvLogic at comparable parameter counts, we introduce refinements within the LGN-ViT framework, reusing its main components to obtain LGN-ViT-SR.

We compare against the published ConvLogic S/M results and our custom width configurations k=9/18/27, evaluated using the ConvLogic reproduction code. The five LGN-ViT-SR configurations correspond, in table order, to ConvLogic k=9, k=18, k=27, S, and M at comparable parameter counts.

ModelParameters (M)CIFAR-10 Test Acc. (%)
LGN-ViT-SR D1080.37157.62
LGN-ViT-SR D1620.73261.82
LGN-ViT-SR D2071.12164.28
LGN-ViT-SR D2281.33064.42
LGN-ViT-SR D69910.66368.93
Original LGN-ViT D7681.47852.56
Original LGN-ViT D595211.45257.75
ConvLogic, custom k=90.37648.76
ConvLogic, custom k=180.75253.08
ConvLogic, custom k=271.12858.36
ConvLogic S1.33760.38
ConvLogic M10.69571.01

LGN-ViT-SR achieves much higher accuracy than the original model. At comparable parameter counts, it exceeds our custom ConvLogic width configurations and the published ConvLogic S in accuracy, while narrowing the original model’s gap to ConvLogic M at the larger scale. Accuracy also increases consistently with parameter count across the SR series. These results show that refinements within the LGN-ViT framework can substantially improve classification accuracy, which further supports the effectiveness of its core architecture.

The refinements were guided by our finding that the original encoder learns useful binary spatial features that the classification readout does not fully use. We therefore encode regional feature activation densities as binary inputs, allowing logic gates to use spatial distributions directly. Following Geva et al. (2021), we also organize the FFN as a key–value memory using existing logic components, explicitly linking binary pattern detection to feature updates. These refinements stay within the LGN-ViT framework and reuse its main logic components.

The methods, experimental settings, and full results are provided in Appendix B.4 of the revised PDF (pp. 22–26), with architecture diagrams in Figures 6–7 and results in Table 13.

Our work establishes the computational basis for expressing input-dependent attention with Boolean operations and integrating it into a trainable Transformer. This extends trainable logic gate networks from feed-forward, convolutional, and recurrent architectures to Transformers. The new results provide further empirical support for this approach and a foundation for further engineering optimization and extension to a broader range of tasks. We thank you again for your constructive feedback and will respond to the remaining questions.

AuthorsOpenReview ↗

Additional CIFAR-100 Results at Comparable Parameter Counts

We sincerely thank the reviewers for suggesting that we evaluate LGN-ViT on more challenging benchmarks. We initially chose MNIST and CIFAR-10 to compare with the results available in the ConvLogic paper (Petersen et al., NeurIPS 2024). We agree that broader evaluation is important and have completed additional CIFAR-100 experiments since our previous comments.

Across the four evaluated configurations with comparable parameter counts, both LGN-ViT-SR and LGN-ViT with a key-value memory FFN outperform our ConvLogic reproduction. The original LGN-ViT also achieves comparable or higher accuracy. Within each model group in the table, the four rows correspond, in order, to ConvLogic k=9, 18, 27, and 32.

ModelParameters (M)CIFAR-100 Test Acc. (%)
LGN-ViT-SR D1680.77926.62
LGN-ViT-SR D2491.55830.55
LGN-ViT-SR D3122.34632.14
LGN-ViT-SR D3422.77632.73
LGN-ViT + key-value memory FFN D1590.76919.65
LGN-ViT + key-value memory FFN D2401.54924.13
LGN-ViT + key-value memory FFN D3032.33827.73
LGN-ViT + key-value memory FFN D3332.76927.79
Original LGN-ViT D5730.78717.79
Original LGN-ViT D11491.57921.22
Original LGN-ViT D17252.37022.66
Original LGN-ViT D20462.81124.48
ConvLogic, custom k=90.79113.58
ConvLogic, custom k=181.58118.46
ConvLogic, custom k=272.37223.49
ConvLogic, custom k=322.81125.01

LGN-ViT-SR uses the spatial readout and key-value memory FFN, both of which are described in Appendix B.4 of the revised PDF (pp. 22–26). It achieves higher accuracy in all four comparisons, exceeding ConvLogic by 7.72–13.04 percentage points. These results show that the gains from refinements within LGN-ViT extend to the more challenging CIFAR-100 classification task.

For LGN-ViT, we replaced only the original Logic FFN with the key-value memory FFN and trained from scratch at comparable parameter counts. These models also outperform ConvLogic in all four comparisons, by 2.78–6.07 percentage points, further supporting the effectiveness of this FFN design within the original framework.

Furthermore, the original LGN-ViT, trained from scratch with its original FFN and five encoder blocks, outperforms ConvLogic in two comparisons and is within 1 percentage point in the other two. These experiments therefore also directly support the competitiveness of the original LGN-ViT architecture on CIFAR-100.

For these comparisons, we reproduced ConvLogic from the paper's description, as we could not find a publicly available official implementation and the paper does not report CIFAR-100 results. To check our reproduction, we evaluated the corresponding configurations on CIFAR-10 and obtained accuracies within 2 percentage points of the published results. This supports the reliability of our baseline implementation.

These results extend our earlier empirical evidence to CIFAR-100 and further support the effectiveness of the core LGN-ViT architecture. The original architecture is already competitive, while refinements to feature readout and the FFN within the same framework deliver further accuracy gains. These findings provide stronger empirical support for further research on logic-native Transformers. Thank you again for your constructive suggestions, which motivated these additional experiments.

AuthorsOpenReview ↗

Additional Tiny ImageNet Results at Comparable Parameter Counts

Following our CIFAR-100 experiments, we have completed additional experiments on the 200-class Tiny ImageNet dataset. Across the four evaluated configurations with comparable parameter counts, both LGN-ViT-SR and the original LGN-ViT achieve higher accuracy than our ConvLogic reproduction. Within each model group, the four rows correspond, in order, to ConvLogic k=9, 18, 27, and 32.

ModelParameters (M)Tiny ImageNet Top-1 Acc. (%)
LGN-ViT-SR D1680.78210.82
LGN-ViT-SR D2491.55813.32
LGN-ViT-SR D3122.34914.50
LGN-ViT-SR D3422.77915.10
Original LGN-ViT D5340.7885.80
Original LGN-ViT D10711.5798.63
Original LGN-ViT D16082.3719.20
Original LGN-ViT D19052.8089.73
ConvLogic, custom k=90.7914.28
ConvLogic, custom k=181.5816.64
ConvLogic, custom k=272.3728.92
ConvLogic, custom k=322.8119.51

LGN-ViT-SR outperforms ConvLogic in all four comparisons by 5.58–6.68 percentage points. These results show that the accuracy gains from the previously introduced refinements within the LGN-ViT framework extend to Tiny ImageNet.

Furthermore, the original LGN-ViT, trained from scratch with its original FFN and five encoder blocks, outperforms ConvLogic in all four comparisons by 0.22–1.99 percentage points. These results directly support the competitiveness of the original LGN-ViT architecture on Tiny ImageNet. Both LGN-ViT-SR and the original LGN-ViT show an upward trend in accuracy as parameter count increases.

We retain the main architecture configurations and training settings from the CIFAR-100 experiments, with adaptations for the dataset and comparable parameter budgets. All images are resized to 32×32, and the official training set is split into 90,000 training and 10,000 internal validation images, retaining the 9:1 ratio used in our CIFAR experiments. We select checkpoints on the internal validation set and report final accuracy on the 10,000 labeled official validation images, using the same split sizes and final evaluation set as Nam et al. (ICML 2022, Appendix B.1).

By retaining the existing configurations and limiting additional tuning, these experiments extend our empirical evidence to Tiny ImageNet. Both the original LGN-ViT and LGN-ViT-SR show competitive performance, further supporting the core architecture and the applicability of its refinements across datasets. We deeply appreciate the time and care the reviewers have devoted to this work. Your thorough assessment and constructive suggestions have helped us more fully demonstrate the value of LGN-ViT's architecture and its competitiveness across datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.