acceptodds
Under review as a conference paper at ICLR 2027

Omni TM-AE: A Scalable and Interpretable Embedding Model Using the Full Tsetlin Machine State Space

Abstract

The increasing complexity of large-scale language models has amplified concerns regarding their interpretability and reusability. While traditional embedding models like Word2Vec and GloVe offer scalability, they lack transparency and often behave as black boxes. Conversely, interpretable models such as the Tsetlin Machine (TM) have shown promise in constructing explainable learning systems, though they previously faced limitations in scalability and reusability. In this paper, we introduce the Omni Tsetlin Machine Autoencoder (Omni TM-AE), a novel embedding model that fully exploits the information contained in the TM's state matrix, including literals previously excluded from clause formation. This method enables the construction of reusable, interpretable embeddings through a single training phase. We further introduce the Tsetlin State-Space Explorer (TSSE), an interactive interpretability tool for analyzing clause-level state distributions and tracing literal-state changes through training, linking learned representations to clauses, feedback events, and supporting training evidence. Extensive experiments across semantic similarity, sentiment classification, word analogy, document clustering, and quantitative interpretability evaluations show that Omni TM-AE performs competitively with mainstream embedding models while providing vocabulary-level traceability. These results demonstrate that it is possible to balance performance, scalability, and interpretability in Natural Language Processing (NLP) systems without resorting to opaque architectures.

Primary area:
interpretability and explainable AI
Keywords:
Tsetlin Machine, word embedding, interpretable AI, representation learning, natural language processing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cUKtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes Omni TM-AE, a Tsetlin Machine-based word embedding technique that leverages the full automaton state space (both included and excluded literals). Embeddings are formed by aggregating state differences between original and negated literals across positive-weighted clauses, creating reusable representations without extra training. The authors also introduce the Tsetlin State-Space Explorer (TSSE) for detailed model inspection. Extensive evaluations across multiple NLP tasks demonstrate that leveraging excluded and negated states is highly informative, allowing Omni TM-AE to achieve competitive performance against conventional embedding methods.

Critical strengths

The paper offers a clear, training-free method for extracting reusable word embeddings directly from a Tsetlin Machine state matrix. While experiments confirm its feasibility for NLP tasks like semantic similarity and clustering, they fail to substantiate the claimed benefits in scalability and interpretability. Nevertheless, the extraction technique remains a practical and useful extension of current Tsetlin Machine embeddings.

Critical weaknesses
  1. Insufficient validation of the core hypothesis concerning excluded literal states. The authors posit that excluded literal states harbor semantic information critical to enhancing embedding quality. However, the ablation study in Table 6 fails to adequately isolate this variable. Given that a vast majority of literal states remain below the inclusion threshold, as depicted in Figure 6, discarding excluded literals simultaneously strips away a massive volume of representational data. This issue is particularly evident when working with large vocabularies. Thus, the ensuing performance drop might stem simply from diminished information coverage rather than the intrinsic semantic utility of the excluded states. To conclusively validate this central claim, the authors must incorporate strictly controlled baselines, such as shuffled state models that maintain the original quantity and distribution of retained states.

  2. Unsubstantiated claims regarding scalability and interpretability. The proposed framework generates embeddings with exceptionally high dimensions matching the vocabulary size (for example, 40,000 dimensions). This stands in stark contrast to the representations of 100 dimensions used by dense baselines. Despite this discrepancy, the manuscript lacks a rigorous comparative analysis of overall computational overhead, memory footprints, or performance and efficiency tradeoffs. Furthermore, the interpretability assessment juxtaposes intrinsic vocabulary aligned explanations against nearest neighbor rationales from dense models. This asymmetry obscures whether the proposed method genuinely offers superior interpretability. Although the embeddings are mathematically traceable to automaton states, this structural transparency does not inherently guarantee that the derived explanations are semantically faithful or meaningful. These methodological shortcomings significantly undermine the overarching claim of striking an optimal balance among scalability, embedding quality, and interpretability.

Questions for authors
  1. Can you report end-to-end training time, peak GPU memory, extraction time, and storage costs across different vocabulary sizes. How do these metrics scale, and how do they compare with dense embeddings under matched resource budgets?

  2. Why is the IMDb perturbation conditioned on ground-truth labels? Please repeat the experiment using a uniform, label-independent perturbation across all examples.

1 comment

Response to Reviewer cUKt

Thank you for the review. We ran the control and the measurements you asked for.

W1. Do the excluded states carry information, or is it only coverage? We ran the shuffled-state control you proposed. In every clause the states of the excluded literals are randomly permuted among the excluded literals. The number of states and their distribution stay exactly the same; only the word each state belongs to is lost. Average Spearman over the five similarity benchmarks, 5 runs:

Embedding built fromAvg. Spearman
all states (paper).540 ± .002
included literals only (Table 6, "w/o Excluded").232 ± .005
excluded literals only.536 ± .002
excluded literals, states shuffled inside each clause.067 ± .024

The excluded states alone reproduce the full score, and the same states attached to the wrong words give almost nothing. The gain comes from what the excluded states encode, not from the amount of data.

W2, Q1. Cost and dimension. One TM-AE (32 clauses) is trained per target word, independently of all other words, on CPU. No GPU memory is used.

VocabularySeconds per word (1 core)CPU-hours, all wordsStates per wordFinal table
5,0000.861.20.32 MB25 MB
10,0001.163.20.64 MB100 MB
20,0001.779.81.28 MB400 MB
40,0003.2135.72.56 MB1.6 GB

End to end for 40,000 words: 31 minutes on 170 cores; about 40 MB per worker plus one shared 3 GB copy of the corpus matrix; reading an embedding from the states takes 0.005 s. The time per word grows linearly with the vocabulary, so the total grows quadratically. On the same machine Word2Vec took 10 CPU-hours, FastText 19, and a PPMI table 3 minutes. So Omni TM-AE costs 4–9× the CPU time of Word2Vec and parallelises trivially over words. The 40,000 dimensions are not needed for quality: a 100-d SVD of the table gives .638 on similarity (Word2Vec .555), and keeping the 1,000 largest values per word gives .630 (120 MB). We will report these trade-offs and scope the word "scalable" to TM-based embeddings.

W2. Interpretability: intrinsic explanations against nearest neighbours. We added a symmetric comparison. A PPMI table built from the same documents is also vocabulary-aligned, so it goes through exactly the procedure used for Omni TM-AE. We also improved how the trained states are read (no retraining): s(a,b)=(ea[b]−mb)+(eb[a]−ma)s(a,b)=(e_a[b]-m_b)+(e_b[a]-m_a), with mjm_j the mean of coordinate jj over target words.

Similarity (avg. Spearman)Feature precision@3, McRae / SWOW
Omni TM-AE, paper read-out.54020.7 / 38.3
Omni TM-AE, improved read-out.63330.4 / 47.3
PPMI, same procedure.64231.5 / 40.5
Word2Vec / FastText / GloVe, nearest neighbours.555 / .551 / .51914.5–15.7 / 34.1–34.9

Precision against human norms is therefore a property of vocabulary-aligned tables in general, and Omni TM-AE is on par with PPMI. Faithfulness mostly reflects how concentrated the scores are (PPMI 10–19, Omni TM-AE 12–20 with the improved read-out), so we will no longer present it as evidence of superior interpretability. We agree that traceability to automaton states does not by itself make an explanation meaningful; the precision against human norms is the evidence we keep, and the claim will be "on par with PPMI, with a logic-based learner", not "optimal balance".

Q2. IMDb with a label-independent perturbation. Done. Two facts first: only the training reviews are perturbed (the test set is untouched), and every row uses the same bag-of-words classifiers; the embedding only chooses the replacement words. The original rule replaced words by close neighbours in positive reviews and by distant words in negative reviews; we agree that tying it to the label makes the result hard to interpret. In the new runs, in every training review, whatever its label, 5% of the tokens are replaced by one of the word's five nearest words. Mean accuracy of the six classifiers of Table 2 over 5 runs:

Replacement words chosen withLabel-independent (new)Label-conditioned (Table 2)
no perturbation.846
Word2Vec.846.841
FastText.846.741
GloVe.846.813
BERT.847.829
PPMI.848
Omni TM-AE.848.838

With the uniform perturbation all sources are within .003 of each other and of the unperturbed reference. The differences in Table 2 came from the label-conditioned rule, not from embedding quality. We will replace Table 2 by this table and no longer use the experiment as evidence of embedding quality.

Reviewer_YVE4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Omni TM-AE, a word embedding method built on the Tsetlin Machine autoencoder. Prior TM embedding work only used literals whose automaton states pass the inclusion threshold. Here, the embedding for a target word is read from the full state matrix: for each vocabulary feature, the state of the negated literal is subtracted from the state of the original literal, summed over positive-weighted clauses, and normalized. The result is a d-dimensional, vocabulary-aligned vector obtained in a single training phase. The authors also present TSSE, a tool for inspecting literal states and tracing training history. Evaluation covers word similarity (5 benchmarks), IMDb sentiment classification under a perturbation protocol, Google analogy, word- and document-level clustering, an interpretability evaluation against feature-norm datasets, and an ablation of the extraction formula.

Critical strengths
  • The core idea is simple and, as far as I know, new for TMs. Using sub-threshold automaton states instead of discarding them removes the retraining step required by Kadhim et al. (2025). The ablation (Table 6) supports that excluded literals and the signed difference carry useful signal, which is the paper’s main claim.

  • The representation is vocabulary-aligned and each coordinate can be traced back to clause states and training events. Combined with reasonable similarity results and the strong semantic analogy numbers (0.465 vs. 0.435 for Word2Vec), this is a credible step toward making TM-based embeddings practical.

Critical weaknesses
  • I don’t think the comparisons support the interpretability and competitiveness claims as currently framed. Omni TM-AE produces 40k-dimensional, vocabulary-aligned vectors, but it is only compared with 100-dim dense models. The natural baselines are missing: count-based vectors such as PPMI/SVD, and sparse interpretable embeddings (e.g., NNSE, SPINE). These are equally “vocabulary-aligned” and would show whether the TM machinery adds anything beyond document co-occurrence. The interpretability protocol also seems to favor the method structurally. Omni TM-AE explanations are its own coordinates, while baselines use nearest neighbors. Faithfulness measured by removing top features is therefore close to tautological for a coordinate-aligned model. The Lancaster result (P@3 76.1 ± 0.0, faithfulness 1.0) also suggests that something about the protocol isn’t working as expected.

  • The scalability and efficiency claims are not backed by actual cost numbers. The text says the embedding is obtained “by training TM-AE for the target word,” and the RG65 comparison reports 47 seconds “from the pre-generated representation.” However, I could not find the total training time or memory cost for producing embeddings for the full 40k vocabulary, nor how that scales with vocabulary size. Since scalability is the main motivation over prior TM work, this needs to be stated explicitly. There are also internal inconsistencies that make the results hard to trust. The “Full” similarity Spearman in Table 6 (0.466) does not match Table 1 (0.536). The “w/o Signed Sub.” variant reaches −0.412, which is a strongly negative correlation and needs an explanation.

Questions for authors
  1. What is the end-to-end training cost (wall time, memory) to produce embeddings for the whole 40k vocabulary? Is one TM-AE trained per target word, or are words trained jointly as CoTM outputs?
  2. Could you add a PPMI (or PPMI+SVD) baseline built on the same binarized document co-occurrence data, and ideally one sparse interpretable embedding baseline? This would most affect my assessment.
  3. How exactly is faithfulness computed for the dense baselines vs. Omni TM-AE, and why is Lancaster faithfulness near zero with zero variance in P@k?
  4. Why does Table 6 “Full” differ from Table 1? Why does removing the signed subtraction give a strongly negative correlation rather than one near zero?
  5. In the sentiment experiment, how are word embeddings turned into document features, especially for the 40k-dim Omni vectors and for BERT? The label-conditioned perturbation uses each model’s own neighbors, so what does this experiment measure beyond standard classification?
  6. The baseline numbers look low for 1B-word training (e.g., Word2Vec MEN 0.577). Was the 40k vocabulary restriction the cause, and were the baselines tuned?
  7. Are the “±x%” values in Tables 1, 2, and 4 relative or absolute standard deviations? If every run uses seed 42, where does the variance across the five runs come from?
1 comment

Response to Reviewer YVE4

Thank you for the review. We ran the experiments you asked for.

Q2. PPMI on the same binarised document data. Spearman; Omni TM-AE is the mean of 5 new runs.

RG65WS353MT287MT771MENAvg.
Omni TM-AE, paper read-out ea[b]e_a[b].625.477.517.483.595.540 ± .002
Omni TM-AE, improved read-out.671.632.542.599.719.633 ± .004
PPMI, PPMI$(a,b)$.662.659.572.588.729.642
Word2Vec / FastText / GloVe (Table 1).555 / .551 / .519

Improved read-out, on the same trained states (no retraining): s(a,b)=(ea[b]−mb)+(eb[a]−ma)s(a,b)=(e_a[b]-m_b)+(e_b[a]-m_a), where mjm_j is the mean of coordinate jj over target words. It removes the frequency of the feature word and uses both trained machines of a pair.

So the TM machinery reaches the PMI level and does not exceed it on similarity. With the same procedure on both tables, Omni TM-AE is ahead on semantic analogy (.585 vs .557) and on feature precision@3 for the SWOW association norms (47.3 vs 40.5), level on McRae (30.4 vs 31.5) and behind on clustering (NMI .358 vs .395). We will change the claims accordingly: the contribution is that a TM learns this information in its sub-threshold states (clause literals only: .23) in one training phase. PPMI rows (85.5% zeros, one dimension per word) are also the sparse interpretable baseline.

Q1. Cost. One TM-AE is trained per target word, independently of the others (not jointly as CoTM outputs), on CPU. With 40,000 words: 3.2 s per word on one core (linear in the vocabulary size), 35.7 CPU-hours for the vocabulary, 31 minutes on 170 cores, 2.56 MB of states per word. The final table is 1.6 GB, or 16 MB after a 100-d SVD with the same similarity (.638). On the same machine Word2Vec needs 10 CPU-hours and PPMI 3 minutes. We will report this and scope "scalable" to TM-based embeddings.

Q3. Faithfulness and Lancaster. One formula for every method, applied to its scores of the candidate words: the share of positive true-feature evidence held by the top-4 explanation words minus the share held by 4 random words. You are right that it favours a coordinate-aligned model: it is high when few scores are positive. PPMI under the same procedure gets 18.5 / 10.1 (McRae / SWOW), Omni TM-AE with the improved read-out 19.7 / 11.9, the paper's read-out 47 / 76, dense models 6–11. We will rename it and drop the claim. Lancaster offers only 11 candidate words, six of them true for every concept; their coordinates are ≤ 0 for 97.7% of the concepts, so there is no positive evidence and faithfulness is ≈ 1%. The "± 0.0" is rounding (17,085 concepts). The set is not informative and will be removed.

Q4. Same model. Table 6 averaged six benchmarks (SimLex-999 included) and one run whose states were not saved correctly; on the five benchmarks of Table 1 its "Full" value is .538 (Table 1: .536). The negative value: without the subtraction each vector is almost exactly the negative of the full one (correlation −0.99 on test words), because with 40,000 words the negated-literal states carry nearly all the variance. A score read from one coordinate flips sign (−.469 against +.538); cosine-based analogy does not change.

Q5. Sentiment. In every row the documents are bag-of-words counts and the classifiers are the same; the embedding (also BERT) only chooses the replacement words in the training reviews, and the test set is untouched. Because the rule was label-conditioned, we reran it (5 runs, mean accuracy of the six classifiers of Table 2). No perturbation: .846. Label-independent perturbation (5% of the tokens of every training review replaced by one of the word's five nearest words): .846–.848 for all six sources (Word2Vec, FastText, GloVe, BERT, PPMI, Omni TM-AE). The sources cannot be told apart, so we will replace Table 2 by these numbers and not use this experiment as evidence of embedding quality.

Q6. Baseline numbers. The cause is the input, not the vocabulary or tuning (defaults, App. A.4). To give every model the same information, the baselines were trained on the same binarised documents as Omni TM-AE: each sentence as the set of its vocabulary words, without word order or repeats (Section 3). With the original word order restored (same cleaning and vocabulary, 100-d, 5 epochs) Word2Vec reaches .653 (MEN .681). With word order a window model is thus slightly above the two document-level methods (Omni TM-AE .633, PPMI .642). We will report both settings.

Q7. The ± values in Tables 1, 2 and 4 are relative (standard deviation divided by the mean, in %); we will switch to the plain standard deviation. Seed 42 fixes the classifiers, k-means and t-SNE; the embeddings differed between runs because the TM was not seeded and Word2Vec/FastText ran on 4 threads.

Reviewer_jzaPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds word embeddings from a Tsetlin Machine autoencoder (TM-AE). Earlier TM-AE embeddings used only the included literals of each clause, i.e., those whose automaton state exceeds the threshold N. Omni TM-AE instead defines the embedding of a target word as follows: for every vocabulary feature x_i, sum over positive-weighted clauses the difference between the automaton states of x_i and of its negation ¬x_i, then normalize. Each embedding dimension therefore corresponds to a vocabulary word. The authors argue that this produces reusable embeddings in a single training phase, unlike prior TM-based approaches. They also present TSSE, an interactive tool for inspecting literal states and tracing their training history.

The evaluation covers word similarity (5 benchmarks), IMDb sentiment classification under a label-conditioned perturbation, Google analogies, document clustering on 5 datasets, an interpretability evaluation against semantic feature norms, and an ablation of the embedding extraction. Results are roughly on par with or below 100-d Word2Vec/GloVe/FastText trained on the One Billion Word benchmark.

Critical strengths
  1. The observation that sub-threshold automaton states carry information is sensible, and the ablation (Table 6) supports it within the TM family: removing excluded literals drops Spearman from 0.47 to 0.19.
  2. Making TM-based embeddings reusable without a second training phase is a practical improvement over Kadhim et al. (2025) for the TM community.
  3. Vocabulary-aligned dimensions, together with a tool that traces state changes back to the training documents, is an appealing direction for data attribution.
  4. Code is provided as supplementary material.
Critical weaknesses

W1. The most relevant baselines are missing: count-based and sparse interpretable embeddings. Omni TM-AE produces a d-dimensional (here 40,000-d), vocabulary-aligned vector from binary document-level co-occurrence (App. A.3: "word frequencies or co-occurrence statistics within documents are ignored"; input X is document presence/absence). The obvious comparison is a document-level PPMI (or log-odds) co-occurrence vector over the same vocabulary. It is equally interpretable (one dimension per word), needs no training, and is well known to be competitive with Word2Vec on word similarity at this scale. Without it, the paper cannot show that the Tsetlin dynamics add anything beyond a signed co-occurrence statistic. Indeed, App. A.6 explains a cluster anomaly ("partner" near "network") precisely through raw document co-occurrence counts. The established line of interpretable sparse word embeddings is also neither discussed nor compared: NNSE (Murphy et al., 2012), sparse overcomplete vectors (Faruqui et al., 2015), SPINE (Subramanian et al., 2018), Word2Sense (Panigrahi et al., 2019) and POLAR (Mathew et al., 2020).

W2. The interpretability evaluation favors Omni TM-AE by construction. For Omni TM-AE, "explanations are obtained directly from vocabulary-aligned literal states, while dense baselines use nearest-neighbor evidence" (§3.5). Faithfulness is measured by removing the top-ranked explanatory features. For Omni TM-AE, removing a feature zeroes a coordinate of the very vector being evaluated. For the dense baselines, the "explanations" are other words (nearest neighbors), and removing them does not change the evaluated vector at all. The large faithfulness gaps (e.g., 75.0 vs. 6.0 on SWOW) therefore reflect the definition of the metric, not a property of the learned representation. Precision against feature norms has a similar problem: a vocabulary-indexed co-occurrence profile is naturally aligned with association norms such as SWOW and Lancaster. This again calls for a PPMI baseline evaluated with the same explanation procedure. Note also that on intrusion, where explanations are compared more directly, the dense baselines are at or near ceiling (99–100%) and Omni TM-AE is lower on 3 of 4 datasets. On Lancaster, Omni TM-AE's faithfulness is 1.0%, which goes unexplained.

W3. The sentiment-classification protocol is flawed. Before classification, 5% of the words in each document are replaced in a label-conditioned way, using each model's own embedding. Positive documents receive one of the 5 nearest neighbors, negative documents one of the 5 least similar among the 400 nearest (§3.2, App. A.4). The perturbation thus injects a label-dependent signal whose strength depends on the geometry of the evaluated embedding. Accuracy measures how detectable this artifact is, not how good the embedding is for sentiment. It is also unclear how documents are represented: averaged vectors? Omni TM-AE uses 20,000-d versus 100-d for the others. Finally, a fine-tuned BERT reaching 0.83 on IMDb (it normally reaches ~0.94) indicates that the BERT pipeline is either misconfigured or not fine-tuned end-to-end, so this row is not a meaningful reference.

W4. The empirical results do not support the claims of competitiveness and advancement. On word similarity, Omni TM-AE's average (0.536) is below Word2Vec (0.555) and FastText (0.551), using 40,000 dimensions against 100. It is far below the off-the-shelf models in App. A.7 (~0.70–0.80). On clustering, it is the worst method on average in both NMI and ARI (Table 4), and its only "win" (Yelp NMI) comes with ARI ≤ 0.006 for every method, over 11,537 labels. NMI with that many classes is not a meaningful measure of clustering quality. On analogies, syntactic accuracy is the lowest of all methods, and the semantic advantage comes mostly from the capital/nationality categories. A conclusion that "Omni TM-AE marks a substantial advancement in bridging the gap between interpretable learning and high-performing NLP embeddings" is not supported.

W5. Inconsistencies between the main results and the ablation. The "Full" ablation row reports Spearman 0.466 ± 0.186 and semantic analogy 0.368. The main tables report 0.536 (Table 1) and 0.4652 (Table 3) for what appears to be the same model. Removing the signed subtraction yields a strongly negative Spearman (−0.412) while leaving analogy accuracy almost unchanged (0.291 vs. 0.297). A strongly anti-correlated similarity suggests that without subtraction the representation is dominated by an artifact, for example the negated-literal states, which App. A.2 says dominate at large vocabularies. This deserves an explanation, since it bears directly on what the embedding actually encodes.

W6. The scalability claims are not quantified. "Scalable" is a headline claim, but the paper does not report the total training time and memory needed to produce embeddings for the full 40k vocabulary, compared with Word2Vec/GloVe on the same hardware. The only number given is "47 seconds from the pre-generated representation" for RG65. It is also unclear whether one CoTM is trained per target word or jointly, and how the cost scales with vocabulary size (the state matrix is clauses × 2d per target). The prior TM baselines are evaluated on one dataset only (TM-AE on RG65, "not conducted due to time constraints"), and Kadhim et al. (2025) is never evaluated.

W7. The interpretability of a dimension is asserted, not demonstrated. A coordinate is a normalized signed sum of automaton states, i.e., a quantity shaped by Type I/II feedback probabilities, clause sharing and the specificity s. It is not obvious why this is more interpretable than a PPMI value, whose meaning is precise. Fig. 3 actually shows that the only included literal of a "happy" clause is "fox", and that antonyms ("sad", "angry") sit at similar states as "smile"/"delightful". This suggests that the states encode topical co-occurrence, not semantics. A user study, or a task-grounded interpretability evaluation (e.g., debugging a downstream error), would be needed to support the claim. The TSSE tool may be useful in practice, but a visualization tool is not by itself a research contribution at ICLR.

W8. Presentation.

  • Figure 3 contains screenshot artifacts ("Epoch [object HTMLInputElement]", "Loading [MathJax]/extensions/MathMenu.js").
  • Tables 1, 2, 4 and 5 are hard to parse. The standard-deviation rows are detached from their entries, and in Table 5 the dataset and method labels are misaligned.
  • The normalization t_i = 2|C| is the same for every i, so the "count" construction is a per-word constant that does not affect cosine similarity. Presenting it as a component is misleading.
  • The related work (App. A.1) describes GloVe as "simple statistical methods … counting the frequency of word occurrences" and Word2Vec/FastText as "black-box", without engaging with the literature that links these models to PMI factorization. That literature is precisely what makes count-based vectors the natural comparison (W1).

Justification. Within the Tsetlin Machine line of work, using the full state space is a sensible engineering step. For ICLR, however, three problems are decisive. The paper omits the most relevant baselines (PPMI co-occurrence vectors and sparse interpretable embeddings). Its interpretability metrics favor the proposed method by construction. Its sentiment experiment uses a label-conditioned perturbation that makes the results uninterpretable. Even taken at face value, the results do not exceed 2013-era 100-d embeddings while using 400× more dimensions. The paper would fit better at a venue focused on Tsetlin Machines, after the baselines and evaluation are fixed.

Questions for authors
  1. How does a document-level PPMI or log-odds co-occurrence vector over the same 40k vocabulary perform on all your tasks? How do the interpretability metrics come out when PPMI explanations are produced by the same procedure as for Omni TM-AE?
  2. How exactly is faithfulness computed for the dense baselines, and does removing nearest-neighbor "explanations" change the evaluated vector in any way?
  3. Why is the IMDb perturbation label-conditioned, and what are the results without any perturbation? How is BERT trained and used in Table 2?
  4. Please explain the gap between the ablation "Full" row and Tables 1 and 3, and the −0.412 Spearman of the variant without signed subtraction.
  5. What are the end-to-end wall-clock time and memory to embed the full vocabulary, compared with Word2Vec/GloVe on the same machine?
  6. Why does Omni TM-AE achieve only 1.0% faithfulness on Lancaster?
2 comments

Response to Reviewer jzaP (1/2)

Thank you for the detailed review. We ran every experiment you asked for. In short:

  • PPMI on the same binary document matrix reaches .642 on word similarity; Omni TM-AE reaches .540 with the paper's read-out and .633 with an improved read-out of the same trained states. It is on par with PPMI: not above it on similarity, ahead on semantic analogy and on the SWOW association norms, behind on clustering.
  • Improved read-out (no retraining): s(a,b)=(ea[b]−mb)+(eb[a]−ma)s(a,b)=(e_a[b]-m_b)+(e_b[a]-m_a), where mjm_j is the mean of coordinate jj over target words. It removes the frequency of the feature word and uses both trained machines of a pair.

W1, Q1. PPMI on all tasks. Spearman; Omni TM-AE is the mean of 5 new runs.

RG65WS353MT287MT771MENAvg.
Omni TM-AE, paper read-out ea[b]e_a[b].625.477.517.483.595.540 ± .002
Omni TM-AE, improved read-out.671.632.542.599.719.633 ± .004
PPMI, PPMI$(a,b)$.662.659.572.588.729.642
Word2Vec / FastText / GloVe (Table 1), cosine.555 / .551 / .519

The same procedure applied to the Omni TM-AE table (mean of two full tables) and to the PPMI table, with the same questions and pipeline for all models (centred rows: row and column means removed, table averaged with its transpose):

Omni TM-AEPPMIWord2Vec / FastText / GloVe
Analogy, semantic / syntactic (centred rows).585 / .428.557 / .398.443/.427, .444/.442, .391/.333
Similarity, 100-d SVD of the table, cosine.638.651.555, .551, .519
Clustering NMI / ARI (100-d, 4 datasets).358 / .204.395 / .233.371/.199, .372/.203, .358/.191
Feature precision@3, McRae / SWOW30.4 / 47.331.5 / 40.514.5–15.7 / 34.1–34.9

You were right that this comparison was missing. It shows parity: the Tsetlin states carry the information of a PMI-type statistic and do not beat it on similarity. What the paper contributes is that a Tsetlin machine learns this information, with ±1 updates of 8-bit automata, in the sub-threshold states that earlier TM embeddings discarded (clause literals only: .23), and that reading it needs no second training phase. We will word the claims this way and add PPMI and PPMI-SVD to every table. PPMI rows (85.5% zeros, one dimension per word) also serve as the sparse interpretable baseline; NNSE, sparse overcomplete vectors, SPINE, Word2Sense, POLAR and the PMI-factorisation results will be discussed in the related work.

W2, Q2, Q6. Interpretability metrics. Faithfulness uses one formula for every method, applied to that method's scores of the candidate words (coordinates for Omni TM-AE, cosines for dense models): the share of positive true-feature evidence held by the top-4 explanation words minus the share held by 4 random words. No vector is edited. We still agree with your conclusion: under the identical procedure PPMI gets 18.5 / 10.1 (McRae / SWOW) and Omni TM-AE with the improved read-out 19.7 / 11.9, against 47 / 76 for the paper's read-out and 6–11 for dense models. The metric measures how concentrated the scores are, not a property of the TM. We will rename it and drop the claim; intrusion is at ceiling for every method and will be dropped. Precision (table above) is the informative metric.

Lancaster (Q6). This set offers only 11 candidate words (hearing, taste, …, torso), six of them true for every concept. Their coordinates are ≤ 0 for 97.7% of the concepts, so there is no positive evidence to remove and faithfulness is ≈ 1%. With so few candidates the set is not informative for any method; it will be removed.

W3, Q3. IMDb. The description was too short. Three facts from the submitted code: (i) only the training reviews are perturbed and the test set is untouched, so a label-dependent artefact cannot be detected at test time; (ii) every row uses the same bag-of-words features, embeddings are never classifier inputs, so there is no 20,000-d against 100-d issue; (iii) BERT and ELMo only choose the replacement words: 0.83 is the accuracy of the same bag-of-words classifiers, not of BERT. A label-conditioned rule is still hard to interpret, so we reran the experiment (5 runs, mean accuracy of the six classifiers of Table 2). No perturbation: .846. Label-independent perturbation (in every training review 5% of the tokens are replaced by one of the word's five nearest words): Word2Vec .846, FastText .846, GloVe .846, BERT .847, PPMI .848, Omni TM-AE .848. All sources are within .003 of each other and of the unperturbed reference: the experiment does not separate embeddings. The differences in Table 2 came from the label-conditioned rule, not from embedding quality. We will replace Table 2 by these numbers and no longer use it as evidence of embedding quality.

(continued in part 2/2: W4–W8)

Response to Reviewer jzaP (2/2)

W4. Claims. Agreed. "Substantial advancement" will be replaced by what the tables in part 1 show: on par with PPMI and with the dense models trained on the same input (better on similarity and semantic analogy, level on syntactic analogy and clustering). "Same input" means that the baselines also received each sentence as an unordered set of words. With the original word order restored (same cleaning and vocabulary, 5 epochs) Word2Vec reaches .653, slightly above the two document-level methods (Omni TM-AE .633, PPMI .642). We will report both settings. Yelp will be removed from the clustering table (its 11,537 labels are business identifiers). The App. A.7 models see far more text and are an upper reference only.

W5, Q4. Table 6 versus Tables 1 and 3. It is the same model. Table 6 averaged six benchmarks (it included SimLex-999, where every model is low) and one run whose states were not saved correctly, which also caused its large ± values. On the five benchmarks of Table 1 the ablation "Full" is .538 (Table 1: .536), and its analogy scores are .460 / .319 (Table 3: .465 / .317). The table will be corrected.

The −0.41 without signed subtraction. It is the mirror image of "Full" (corrected values: −.469 against +.538, with the same mirror on every dataset). Without the subtraction each vector is almost exactly the negative of the full one (correlation −0.99 on test words), because with 40,000 words the negated-literal states carry nearly all the variance (App. A.2). A score read from one coordinate therefore flips sign, while cosine-based analogy does not change (.451 against .460). It is not an artefact: the evidence is stored in the "absence" literals, and this is what the embedding encodes.

A new control makes the same point (5 runs, average Spearman): all states .540; clause (included) literals only .232; excluded literals only .536; excluded literals with their states shuffled inside each clause .067.

W6, Q5. Cost. One TM-AE (32 clauses) is trained per target word, independently of all other words, on CPU only.

VocabularySeconds per word (1 core)CPU-hours, all wordsStates per wordFinal table
5,0000.861.20.32 MB25 MB
10,0001.163.20.64 MB100 MB
20,0001.779.81.28 MB400 MB
40,0003.2135.72.56 MB1.6 GB

End to end for 40,000 words: 31 minutes on 170 cores (measured rate: 21 words per second); one complete run on 120 cores shared with other jobs took 44 minutes. Memory: about 40 MB per worker plus one shared 3 GB copy of the corpus matrix. On the same machine Word2Vec took 2.5 h and FastText 4.7 h on 4 threads (10 and 19 CPU-hours), GloVe 28 min on 8 threads, PPMI 3 minutes on one core. So Omni TM-AE costs 4–9× the CPU time of Word2Vec, parallelises trivially over words, and is far more expensive than PPMI.

"Scalable" is meant relative to earlier TM embeddings: TM-AE trains all target words in one machine (about 8 h for the 48 words of RG65, which is why it was run on RG65 only), and Kadhim et al. (2025) needs a second training phase for every new set of words. We will state this instead of "time constraints" and scope the word "scalable" accordingly. The 40,000 dimensions are not needed for quality: a 100-d SVD of the table gives .638, and keeping the 1,000 largest values per word gives .630.

W7. What a dimension means. Coordinate ii is the mean over clauses of (state of xix_i − state of ¬xi\neg x_i)/2: two 8-bit counters that move by ±1 at each feedback event, and TSSE traces each event to its training documents. We do not claim that this is more interpretable than a PPMI value. "fox" is the point of Figure 3: the clause alone describes "happy" poorly, while the sub-threshold states hold the context (control above: .23 against .54). Antonyms: agreed, the model captures relatedness, like any distributional model (SimLex-999: .20; we will report it). No user study was done; we will state this as a limitation. TSSE will be presented as a supporting tool, not as a contribution.

W8. Presentation. The two strings cannot be seen in the PDF at any zoom level. They are clipped objects inside the figure's vector source and appear only when the PDF is passed through automatic text extraction. They will be removed. The tables will be reformatted with each ± next to its value. ti=2∣C∣t_i=2|C| is constant over ii, as you say; it normalises for the number of positive clauses, which differs between target words (17.6 of 32 on average). The related work will be rewritten around the PMI-factorisation view.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.