W1. The most relevant baselines are missing: count-based and sparse interpretable embeddings. Omni TM-AE produces a d-dimensional (here 40,000-d), vocabulary-aligned vector from binary document-level co-occurrence (App. A.3: "word frequencies or co-occurrence statistics within documents are ignored"; input X is document presence/absence). The obvious comparison is a document-level PPMI (or log-odds) co-occurrence vector over the same vocabulary. It is equally interpretable (one dimension per word), needs no training, and is well known to be competitive with Word2Vec on word similarity at this scale. Without it, the paper cannot show that the Tsetlin dynamics add anything beyond a signed co-occurrence statistic. Indeed, App. A.6 explains a cluster anomaly ("partner" near "network") precisely through raw document co-occurrence counts. The established line of interpretable sparse word embeddings is also neither discussed nor compared: NNSE (Murphy et al., 2012), sparse overcomplete vectors (Faruqui et al., 2015), SPINE (Subramanian et al., 2018), Word2Sense (Panigrahi et al., 2019) and POLAR (Mathew et al., 2020).
W2. The interpretability evaluation favors Omni TM-AE by construction. For Omni TM-AE, "explanations are obtained directly from vocabulary-aligned literal states, while dense baselines use nearest-neighbor evidence" (§3.5). Faithfulness is measured by removing the top-ranked explanatory features. For Omni TM-AE, removing a feature zeroes a coordinate of the very vector being evaluated. For the dense baselines, the "explanations" are other words (nearest neighbors), and removing them does not change the evaluated vector at all. The large faithfulness gaps (e.g., 75.0 vs. 6.0 on SWOW) therefore reflect the definition of the metric, not a property of the learned representation. Precision against feature norms has a similar problem: a vocabulary-indexed co-occurrence profile is naturally aligned with association norms such as SWOW and Lancaster. This again calls for a PPMI baseline evaluated with the same explanation procedure. Note also that on intrusion, where explanations are compared more directly, the dense baselines are at or near ceiling (99–100%) and Omni TM-AE is lower on 3 of 4 datasets. On Lancaster, Omni TM-AE's faithfulness is 1.0%, which goes unexplained.
W3. The sentiment-classification protocol is flawed. Before classification, 5% of the words in each document are replaced in a label-conditioned way, using each model's own embedding. Positive documents receive one of the 5 nearest neighbors, negative documents one of the 5 least similar among the 400 nearest (§3.2, App. A.4). The perturbation thus injects a label-dependent signal whose strength depends on the geometry of the evaluated embedding. Accuracy measures how detectable this artifact is, not how good the embedding is for sentiment. It is also unclear how documents are represented: averaged vectors? Omni TM-AE uses 20,000-d versus 100-d for the others. Finally, a fine-tuned BERT reaching 0.83 on IMDb (it normally reaches ~0.94) indicates that the BERT pipeline is either misconfigured or not fine-tuned end-to-end, so this row is not a meaningful reference.
W4. The empirical results do not support the claims of competitiveness and advancement. On word similarity, Omni TM-AE's average (0.536) is below Word2Vec (0.555) and FastText (0.551), using 40,000 dimensions against 100. It is far below the off-the-shelf models in App. A.7 (~0.70–0.80). On clustering, it is the worst method on average in both NMI and ARI (Table 4), and its only "win" (Yelp NMI) comes with ARI ≤ 0.006 for every method, over 11,537 labels. NMI with that many classes is not a meaningful measure of clustering quality. On analogies, syntactic accuracy is the lowest of all methods, and the semantic advantage comes mostly from the capital/nationality categories. A conclusion that "Omni TM-AE marks a substantial advancement in bridging the gap between interpretable learning and high-performing NLP embeddings" is not supported.
W5. Inconsistencies between the main results and the ablation. The "Full" ablation row reports Spearman 0.466 ± 0.186 and semantic analogy 0.368. The main tables report 0.536 (Table 1) and 0.4652 (Table 3) for what appears to be the same model. Removing the signed subtraction yields a strongly negative Spearman (−0.412) while leaving analogy accuracy almost unchanged (0.291 vs. 0.297). A strongly anti-correlated similarity suggests that without subtraction the representation is dominated by an artifact, for example the negated-literal states, which App. A.2 says dominate at large vocabularies. This deserves an explanation, since it bears directly on what the embedding actually encodes.
W6. The scalability claims are not quantified. "Scalable" is a headline claim, but the paper does not report the total training time and memory needed to produce embeddings for the full 40k vocabulary, compared with Word2Vec/GloVe on the same hardware. The only number given is "47 seconds from the pre-generated representation" for RG65. It is also unclear whether one CoTM is trained per target word or jointly, and how the cost scales with vocabulary size (the state matrix is clauses × 2d per target). The prior TM baselines are evaluated on one dataset only (TM-AE on RG65, "not conducted due to time constraints"), and Kadhim et al. (2025) is never evaluated.
W7. The interpretability of a dimension is asserted, not demonstrated. A coordinate is a normalized signed sum of automaton states, i.e., a quantity shaped by Type I/II feedback probabilities, clause sharing and the specificity s. It is not obvious why this is more interpretable than a PPMI value, whose meaning is precise. Fig. 3 actually shows that the only included literal of a "happy" clause is "fox", and that antonyms ("sad", "angry") sit at similar states as "smile"/"delightful". This suggests that the states encode topical co-occurrence, not semantics. A user study, or a task-grounded interpretability evaluation (e.g., debugging a downstream error), would be needed to support the claim. The TSSE tool may be useful in practice, but a visualization tool is not by itself a research contribution at ICLR.
W8. Presentation.
- Figure 3 contains screenshot artifacts ("Epoch [object HTMLInputElement]", "Loading [MathJax]/extensions/MathMenu.js").
- Tables 1, 2, 4 and 5 are hard to parse. The standard-deviation rows are detached from their entries, and in Table 5 the dataset and method labels are misaligned.
- The normalization t_i = 2|C| is the same for every i, so the "count" construction is a per-word constant that does not affect cosine similarity. Presenting it as a component is misleading.
- The related work (App. A.1) describes GloVe as "simple statistical methods … counting the frequency of word occurrences" and Word2Vec/FastText as "black-box", without engaging with the literature that links these models to PMI factorization. That literature is precisely what makes count-based vectors the natural comparison (W1).
Justification. Within the Tsetlin Machine line of work, using the full state space is a sensible engineering step. For ICLR, however, three problems are decisive. The paper omits the most relevant baselines (PPMI co-occurrence vectors and sparse interpretable embeddings). Its interpretability metrics favor the proposed method by construction. Its sentiment experiment uses a label-conditioned perturbation that makes the results uninterpretable. Even taken at face value, the results do not exceed 2013-era 100-d embeddings while using 400× more dimensions. The paper would fit better at a venue focused on Tsetlin Machines, after the baselines and evaluation are fixed.
Response to Reviewer cUKt
Thank you for the review. We ran the control and the measurements you asked for.
W1. Do the excluded states carry information, or is it only coverage? We ran the shuffled-state control you proposed. In every clause the states of the excluded literals are randomly permuted among the excluded literals. The number of states and their distribution stay exactly the same; only the word each state belongs to is lost. Average Spearman over the five similarity benchmarks, 5 runs:
The excluded states alone reproduce the full score, and the same states attached to the wrong words give almost nothing. The gain comes from what the excluded states encode, not from the amount of data.
W2, Q1. Cost and dimension. One TM-AE (32 clauses) is trained per target word, independently of all other words, on CPU. No GPU memory is used.
End to end for 40,000 words: 31 minutes on 170 cores; about 40 MB per worker plus one shared 3 GB copy of the corpus matrix; reading an embedding from the states takes 0.005 s. The time per word grows linearly with the vocabulary, so the total grows quadratically. On the same machine Word2Vec took 10 CPU-hours, FastText 19, and a PPMI table 3 minutes. So Omni TM-AE costs 4–9× the CPU time of Word2Vec and parallelises trivially over words. The 40,000 dimensions are not needed for quality: a 100-d SVD of the table gives .638 on similarity (Word2Vec .555), and keeping the 1,000 largest values per word gives .630 (120 MB). We will report these trade-offs and scope the word "scalable" to TM-based embeddings.
W2. Interpretability: intrinsic explanations against nearest neighbours. We added a symmetric comparison. A PPMI table built from the same documents is also vocabulary-aligned, so it goes through exactly the procedure used for Omni TM-AE. We also improved how the trained states are read (no retraining): s(a,b)=(ea[b]−mb)+(eb[a]−ma), with mj the mean of coordinate j over target words.
Precision against human norms is therefore a property of vocabulary-aligned tables in general, and Omni TM-AE is on par with PPMI. Faithfulness mostly reflects how concentrated the scores are (PPMI 10–19, Omni TM-AE 12–20 with the improved read-out), so we will no longer present it as evidence of superior interpretability. We agree that traceability to automaton states does not by itself make an explanation meaningful; the precision against human norms is the evidence we keep, and the claim will be "on par with PPMI, with a logic-based learner", not "optimal balance".
Q2. IMDb with a label-independent perturbation. Done. Two facts first: only the training reviews are perturbed (the test set is untouched), and every row uses the same bag-of-words classifiers; the embedding only chooses the replacement words. The original rule replaced words by close neighbours in positive reviews and by distant words in negative reviews; we agree that tying it to the label makes the result hard to interpret. In the new runs, in every training review, whatever its label, 5% of the tokens are replaced by one of the word's five nearest words. Mean accuracy of the six classifiers of Table 2 over 5 runs:
With the uniform perturbation all sources are within .003 of each other and of the unperturbed reference. The differences in Table 2 came from the label-conditioned rule, not from embedding quality. We will replace Table 2 by this table and no longer use the experiment as evidence of embedding quality.