acceptodds
Withdrawn from ICLR 2027

FastOmniTMAE: Parallel Clause Learning for Scalable and Hardware-Efficient Tsetlin Machine Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embedding models in natural language processing (NLP) have achieved remarkable success, but their representations often lack interpretability. The Tsetlin Machine (TM) provides an interpretable, logic-based alternative. Omni TM Autoencoder (Omni TM-AE) applies this paradigm to static embeddings, but its training remains slow. In this work, we propose FastOmniTMAE, which replaces sequential training dependencies with a two-stage parallel process of evaluation and update. Across classification, similarity, and clustering tasks, FastOmniTMAE achieves up to 5 faster training while maintaining comparable embedding quality. We further implement FastOmniTMAE as a reusable accelerator on SoC-FPGA platforms, achieving similarity scores of 0.669 on a resource-constrained FPGA and 0.696 on an UltraScale+ SoC. These results demonstrate an efficient and interpretable approach to logic-based embedding training with a small hardware footprint.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Tsetlin Machine, Word Embeddings, Interpretable Machine Learning, Logic-Based Learning, Natural Language Processing, Parallel Training, FPGA Acceleration, Hardware-Efficient Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wTjbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes FastOmniTMAE, a modified Omni TM-AE embedding model that replaces the global class-sum update probability with a clause-local rule intended to remove synchronization and expose clause-level parallelism. It also presents CPU, CUDA, OpenCL, and SoC-FPGA implementations and introduces two evaluation suites covering classification diagnostics, word similarity, clustering, and cross-platform training time. The paper reports up to a 5×5\times training speedup in the classification experiment, approximately 1.2×1.2\times in the word-similarity experiment, and 5.58×5.58\times–7.08×7.08\times FPGA speedups over a single local ARM core. It further argues that GPUs are poorly matched to TM training and that dedicated FPGA logic is a more suitable platform.

The submitted package contains substantial C, CUDA, OpenCL, Python, and RTL code. However, the central comparisons are not currently valid enough to support the conclusions. The classification evaluation has a reachable stale-cache error, exposes the target feature directly in the input, and reports clause-level diagnostics rather than independent downstream classification. The H100 result comes from an implementation with extremely low launch parallelism, so it cannot support a platform-level conclusion about GPU suitability. Finally, the proposed method changes several learning and implementation choices simultaneously without the controls needed to attribute gains to the local probability rule.

Critical strengths

None that currently supports acceptance. The submission contains real engineering work, including C, CUDA, OpenCL, and RTL implementations with clause engines, state buffers, and AXI integration. However, implementation breadth alone does not validate the algorithmic or platform-level conclusions, and the current experimental comparisons do not isolate those conclusions cleanly.

Critical weaknesses
  1. The main embedding-quality evidence is not a valid independent evaluation of the claimed learning improvement.

    The classification experiment in Figure 4 has multiple consequential problems.

    First, the submitted incremental prediction path caches a reverse literal-to-clause index and false-literal counts. This cache is initialized from the automata states on the first prediction, but the training method subsequently updates those states in place without invalidating the cache. In an execution test using the submitted Python training method and recompiled native C code, training changed the automata states while the cache remained marked as initialized. The same input then produced prediction 1 with the stale cache and prediction 0 after only forcing cache reconstruction. Thus, the reported epoch-wise classification path can evaluate clauses that no longer correspond to the current trained state. This does not prove that the entire Figure 4 curve is caused by the bug, but it prevents that curve from supporting claims about training stability or sustained classification superiority until the relevant version is identified or the experiment is rerun.

    Second, the example generator constructs positive examples only from documents containing the target token and negative examples only from documents not containing it, while retaining the target feature in XX. Therefore, for nondegenerate targets, the generated task satisfies X[target]=Y.X[\text{target}] = Y.

    Calling the submitted native generator on 256 synthetic examples confirmed that directly reading this one input bit recovered all 256 labels. This permits correct prediction without learning semantic context.

    Third, the reported precision, recall, and F1 are not obtained by training an independent downstream classifier on frozen embeddings. The benchmark calls per-clause clause_precision and clause_recall, averages the non-NaN clause statistics, and computes their harmonic mean. Training and metric computation receive the same X_train, while the metric functions regenerate examples from the supplied corpus. Consequently, the stated “500 inference examples” do not, from the submitted path, establish an independent held-out embedding-classification evaluation.

    The other quality comparisons do not repair this problem. FastOmniTMAE’s similarity function returns the coordinate of word 2 in word 1’s learned vector, whereas Word2Vec, FastText, and GloVe are evaluated using cosine similarity. The former can be directional: under a synthetic invocation of the submitted function, score(alpha,beta)=9 and score(beta,alpha)=2, while the cosine baseline was symmetric. Different representation-specific scores may be legitimate, but they do not constitute a common geometric evaluation without clear interpretation and additional controls. Moreover, when the original token cache is absent, the submitted baseline path reconstructs “sentences” from sparse row indices, losing word order and repeated-token frequency; two different bags with different original orders both became the same deduplicated sequence. The package does not contain the cache or result snapshot needed to determine whether the reported baseline runs avoided this path.

    Together, these issues mean that the paper has not established that FastOmniTMAE preserves or improves embedding quality under a common, leakage-resistant, independent evaluation.

  2. The algorithmic and hardware comparisons are not designed to identify the claimed mechanism or platform conclusion.

    FastOmniTMAE is presented primarily as removing a global synchronization dependency while preserving the original learning behavior. However, Algorithm 1 replaces the global class sum with a single local contribution wjojw_j o_j, which changes the stochastic learning dynamics rather than merely rescheduling an equivalent computation. In the submitted configurations, clause weights are initialized to fixed signs and are not updated. Negative-weight clauses are skipped, while only positive-weight clauses train. With positive weight 11 and T=20000T=20000, the local update probability is approximately fixed at 0.5:

    • 0.5 for an inactive clause;
    • 0.499975 for an active positive example;
    • 0.500025 for an active negative example.

    A native 512-update test confirmed that weights remained fixed, negative-weight clause states did not change, and positive-weight clause states did. Therefore, a matched fixed-p=0.5p=0.5 control is essential. Without it, the experiments do not show that the proposed local formula is necessary. Nor do they isolate removing synchronization from simultaneously removing weight learning, changing clause gating, altering feedback behavior, and moving execution into different native implementations. The paper also lacks substantive empirical comparison with directly relevant prior asynchronous and local-feedback TM approaches that it cites.

    The GPU conclusion is even less supported. The appendix itself states that each profiled H100 update launches one 64-thread block, which can occupy at most one of 132 SMs. The default submitted hardware runner uses 128 threads and ⌈number of clauses128⌉\left\lceil \frac{\text{number of clauses}}{128} \right\rceil blocks; with 32 clauses, this is still one block, and only 32 threads perform clause work. Target tokens are processed sequentially, and one kernel is launched per example. Under the Table 2 configuration of 44 targets and 8,000 examples per target, this implies 352,000 sequential single-block update launches. This implementation is structurally incapable of using the H100 at meaningful scale. The statement that CUDA exhibits poor performance “despite full GPU utilization” and that the result demonstrates a general mismatch between GPUs and TM logical operations is therefore not justified. At most, the experiment shows that this particular under-parallelized CUDA implementation performs poorly.

    The FPGA results demonstrate execution of a hardware design, but they do not establish broader efficiency. The reported 7.08×7.08\times ZCU104 speedup is relative to one ARM core on the same board; the ZCU104 time of 119.9 seconds remains approximately 2.08×2.08\times the desktop i9 single-thread time of 57.7 seconds. The designs also consume 86.54% and 75% of available BRAM on ZCU104 and Zybo, respectively. No power, energy, timing report, or bitstream artifact is supplied, so “small hardware footprint” and lower-power implications are not supported by the reported DSP count alone.

    Thus, the experiments currently establish only that the submitted FastOmniTMAE implementation outperforms particular baseline implementations under selected configurations. They do not identify the local probability rule as the cause, establish equivalence to the original learning mechanism, or show that GPUs are inherently unsuitable relative to FPGAs.

Questions for authors
  1. Was Figure 4 generated with the submitted incremental clause-evaluation cache implementation? If not, what exact code revision or cache-invalidation fix was used? Do existing run artifacts show that the clause cache was rebuilt after each training update or epoch?

  2. Was the target-token feature masked from XX in the classification experiment? If so, where does that occur in the experimental version? If not, how should the classification result be interpreted given that the submitted generator exposes X[target]=YX[\text{target}]=Y?

  3. Are Figure 4’s precision, recall, and F1 values clause-level diagnostics computed from the training corpus, or results from an independent downstream classifier evaluated on held-out examples? How are the stated 500 inference examples used in the submitted metric path?

  4. Is there an existing fixed-p=0.5p=0.5 control using the same native implementation, training budget, clause gating, and weight behavior? This would materially affect whether the local probability formula, rather than the surrounding implementation changes, explains the result.

  5. Which existing comparison isolates removal of the class sum from removal of weight learning, the wj ≥ 0 gate, feedback changes, and movement of loops into native code?

  6. How does FastOmniTMAE differ empirically from the cited asynchronous TM architecture and alternate local-feedback mechanism? Were either implemented as a matched baseline?

  7. Why should the H100 result support a platform-level conclusion when the profiled and submitted CUDA paths launch only one block for 32 clauses and process target tokens sequentially? Was any implementation with concurrent targets, multiple literal chunks, or enough blocks to occupy the H100 evaluated?

  8. What does “full GPU utilization” mean in the main text, given the appendix’s stated maximum active-SM coverage of approximately 0.758% for the profiled kernel?

  9. Were Word2Vec, FastText, and GloVe trained from the original ordered token sequences, or from sequences reconstructed from sparse row indices? Was tokenized_sentences.pickle present in the runs used for Table 1?

  10. Should the FastOmniTMAE word-pair score be interpreted as directional association rather than vector cosine similarity? What existing common scoring experiment supports direct comparison with the cosine-based baselines?

  11. Which reported memory figures refer to one target’s temporary automata state, and which refer to a deployable full-vocabulary embedding table? Was the claimed 8-bit vector representation actually used and evaluated in the supplied software and hardware paths?

  12. Can the authors provide the exact code revision, configurations, corpus-cache provenance, and existing raw outputs corresponding to Figures 2–5 and Tables 1–2? A version mismatch that resolves the cache and preprocessing issues could materially change my assessment.

Reviewer_UAWNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FastOmniTMAE, a reformulation of Omni TM-AE for training interpretable static word embeddings. It replaces the global weighted clause sum used to determine feedback probabilities with clause-local contributions, enabling independent clause evaluation and updates. The authors evaluate clause-level prediction statistics, five word-similarity datasets, and a qualitative embedding visualization, and provide implementations for CPUs, GPUs, and two SoC-FPGA platforms. The reported CPU speedup over Omni TM-AE is approximately 5.5 times in the classification-style experiment and 1.2 times in the word-similarity experiment. The FPGA implementations accelerate a 44-token workload relative to their respective single-core ARM baselines. The main contribution is an efficiency-oriented learning-rule reformulation and its hardware implementation, with broadly similar but not uniformly improved word-similarity results.

Critical strengths
  1. The paper addresses a concrete training bottleneck in an interpretable embedding model and reports measurable CPU improvements over its direct predecessor. Under the word-similarity configuration, training time decreases from 293.1 s to 242.9 s while average Spearman correlation changes from 0.543 to 0.537 and average Kendall correlation from 0.390 to 0.392. This provides useful evidence of an efficiency-quality trade-off without a large change in average embedding quality, which is a meaningful positive aspect of the submission.

  2. The work goes beyond a software proposal by implementing training on both ZCU104 and Zybo SoC-FPGA platforms. Table 2 reports 7.08 times and 5.58 times speedups over the respective single-core ARM implementations. Together with the supplied implementation package, these results make the work potentially useful to researchers studying logic-based learning on constrained hardware.

Critical weaknesses
  1. Section 3.1.1 should more clearly delimit what Figure 4 demonstrates. The supplied evaluation code aggregates clause-level precision and recall on examples sampled from the training corpus, rather than evaluating a complete classifier on held-out documents or a downstream classifier using the learned embeddings. This is useful as a training diagnostic, but the concluding claim of improved classification accuracy is broader than the demonstrated result. Please clarify the evaluation protocol and scope of this claim.

  2. The cross-hardware evaluation does not sufficiently establish the claimed hardware suitability of the proposed training method. The paper explicitly lists the analysis of GPU inefficiencies and benchmarking across CPUs, GPUs, and FPGAs among its contributions. However, the GPU implementation exhibits a substantial parallelism limitation. Consequently, the reported GPU slowdown does not distinguish limitations of the implementation from limitations inherent to executing TM operations on GPUs. This weakens the architectural interpretation in Sections 2.2.2 and 3.4 and limits what the benchmark establishes about the relative suitability of the evaluated platforms.

Questions for authors
  1. Figure 4 is labeled “Mean clause score,” and the supplied code computes F1 from averaged clause-level precision and recall using examples sampled from the same corpus matrix used for training. Could you confirm that this is the protocol used for Figure 4 and clarify whether the experiment is intended as a clause-learning diagnostic or evidence of classification generalization?

  2. Section 3.1.1 attributes the declining scores of Omni TM-AE to the global class-sum-based feedback mechanism. What evidence isolates this effect from other differences between the implementations? Could you clarify whether the benefit primarily comes from removing the global decay in update probability or from the clause-dependent local rule? A controlled feedback-rule comparison, including a constant-probability variant if available, would help substantiate this explanation.

  3. Sections 2.2.2 and 3.4 attribute the GPU slowdown to an architectural mismatch, while Appendix A.5 identifies insufficient launch parallelism in the profiled implementation. Could you identify the CUDA execution paths and launch configurations used for Table 2 and Appendix A.5, explain how “full GPU utilization” was measured, and clarify whether the architectural conclusion is supported by a comparison that exploits parallelism across independent target words or clauses? If such a comparison is unavailable, would you restrict the conclusion to the evaluated implementations?

Reviewer_UfTeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FastOmniTMAE, a parallelized training procedure for Tsetlin Machine word embeddings, and implements it on CPUs, GPUs, and two SoC-FPGA boards. It reports comparable word-embedding quality to Omni TM-AE, a 1.2x CPU speedup in one matched training comparison, and up to 7.08x speedup on the ZCU104 FPGA relative to that board's ARM CPU. Its evaluation also includes word similarity, classification, and clustering tasks, plus a multi-hardware benchmark.

Critical strengths
  1. The work provides a concrete hardware implementation, including resource utilization and profiling, rather than stopping at a software proposal. This makes its systems contribution tangible.
  2. The evaluation includes several embedding tasks and reports quality alongside runtime, helping assess whether the parallel reformulation preserves utility.
Critical weaknesses
  1. The central efficiency claim is not yet supported by a broad, matched scaling study. The 1.2x CPU speedup is modest, while the larger FPGA ratio is measured against a resource-constrained, single-core ARM baseline on a small workload. This limits the strength of the general scalability conclusion.
  2. The hardware quality comparison uses RG65 (44 pairs), and the reported FPGA scores differ slightly across devices. More representative workloads and repeated quality evaluation are needed to establish that the accelerator preserves embedding quality robustly.
Questions for authors
  1. Can the authors report matched CPU/FPGA training throughput and embedding quality across several vocabulary sizes, corpus sizes, and downstream datasets, including uncertainty over repeated runs?
  2. Is the reported 1.2x CPU gain representative across configurations and corpus sizes, and how does it scale with thread count?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.