acceptodds
Under review as a conference paper at ICLR 2027

Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers

Abstract

Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder–Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2–7.8% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.

Primary area:
optimization
Keywords:
associative algebra, quiver algebra, parameter-preserving matrix multiplication, transformer feed-forward layers, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xZqfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes replacing standard dense matrix multiplication Y=XWY = XW in Transformer feed-forward networks (FFNs) with a sparser bilinear multiplication table derived from graph-based associative unital algebras (QqQ_q and Q2⊗2Q_2^{\otimes 2}). Specifically, features and weights are partitioned into q×qq \times q blocks; while all K×NK \times N weight parameters are retained, each token position pp is assigned a deterministic "row type" τ(p)\tau(p) that only evaluates diagonal blocks and one row of off-diagonal blocks, reducing active MACs by a factor of (2q−1)/q2(2q-1)/q^2 (or 9/169/16 for Q2⊗2Q_2^{\otimes 2}). The authors prove that this multiplication rule achieves the Alder–Strassen lower bound on bilinear rank, benchmark custom Triton kernels on synthetic tensor shapes, and pretrain a single 110M GPT-2 scale model for 12.3B tokens. The resulting algebraic model achieves a 6.2–7.8% end-to-end generation throughput increase, but suffers noticeable quality degradation on all evaluated downstream benchmarks (GSM8K, IFEval, MBPP).

Overall, while viewing matrix multiplication rules through associative algebras is a neat conceptual angle, the paper is empirically weak, lacks comparisons against obvious structured-matrix/low-rank baselines, and leaves serious doubts about its practical utility and internal representation quality. I recommend Weak Reject.

Critical strengths

1.Interesting perspective on bilinear products: Framing sparse block interactions as an associative unital algebra and connecting the active block count to the Alder–Strassen bilinear rank bound is mathematically clean.

2.Custom kernel engineering: The authors implement custom Triton kernels and honestly report both synthetic kernel microbenchmarks and end-to-end decoding throughput, candidly acknowledging the downstream quality drop.

Critical weaknesses

1.Conceptual Overlap with Structured / Low-Rank Layers and Missing Baselines: At its core, the proposed layer is a position-routed block-sparse projection: each token activates a subset of weight blocks, and the full parameter matrix is covered across different token positions. Despite claiming that "keeping all parameters while changing the product" is a unique paradigm, achieving sub-quadratic or reduced GEMM cost while covering the weight space can also be done via low-rank factorizations (LoRA-style low-rank adapters/factorized layers), Monarch matrices (Dao et al., 2022a), block tensor-train, or simply a FLOP-matched narrower dense FFN (e.g., reducing FFN width FF to 9/16F9/16 F). The paper compares against none of these in either speed or quality. Without showing that the algebraic product outperforms a trivially narrower dense FFN or a Monarch/low-rank layer at the same throughput/FLOPs, the 6.8% speedup paired with a 3.4-point drop on IFEval and a 42% relative drop on MBPP (6.6% →\to 3.8%) simply looks like moving down a standard compute–quality trade-off curve.

2.Thin Empirical Validation and Questionable Position-Type Schedule: The end-to-end validation consists of only a single 110M model run. At 110M scale, generative scores on GSM8K (3.18% vs. 2.20%) and MBPP (6.60% vs. 3.80%) are near the noise floor. More importantly, Appendix D.5 (line 1570) reveals that the 110M pretraining run uses the position schedule τ(p)=⌊(p mod 768)/192⌋\tau(p) = \lfloor (p \bmod 768)/192 \rfloor. This assigns 192 consecutive tokens to the exact same row type! Consequently, for any prompt+response shorter than 192 tokens (which covers the majority of QA and Code prompts in Table 5), the model only ever activates Type-0 off-diagonal weights, leaving 75% of the off-diagonal weight rows completely unused during inference and creating severe position-dependent representation shifts across the 192-token boundary.

3.Lack of Mechanistic / Representation Analysis (e.g., SAE or Feature Probes): By setting J2=0J^2 = 0 (or restricting radical interactions), the algebraic product zeroes out specific cross-group feature interactions depending on token position. How does this algebraic restriction affect the features learned inside the FFN? There is no mechanistic investigation—such as Sparse Autoencoder (SAE) feature probing, sub-block effective rank analysis, or activation distribution checks—to see whether this position-dependent routing causes dead features, polysemantic entanglement, or fragmented representations across the qq feature groups.

4.Doubtful Practical Utility in Memory-Bound LLM Regimes: The slogan "Change the Product, Keep the Parameters" is actually a disadvantage for batched autoregressive decoding. In modern LLMs, small-batch decoding is heavily memory-bandwidth bound, not compute-bound. Because the method retains 100% of the dense parameters (K×NK \times N), any batch containing all qq row types still requires loading the entire weight matrix from HBM into SRAM. Thus, saving MACs without reducing parameter memory traffic yields diminishing returns in real-world serving, as reflected by the collapse from 3x–11x synthetic kernel speedups (Table 1) to just 6.2–7.8% end-to-end throughput gain (Table 2).

Questions for authors

1.How does the 110M algebraic model compare against a trivial FLOP-matched dense baseline (i.e., a dense SwiGLU FFN with intermediate width F=916×1536=864F = \frac{9}{16} \times 1536 = 864), or against a rank-matched low-rank / Monarch layer, in both training loss and generation speed?

2.Why did the 110M experiment use a contiguous 192-token block schedule ⌊(p mod 768)/192⌋\lfloor (p \bmod 768)/192 \rfloor (Appendix D.5) instead of the cyclic schedule τ(p)=1+((p−1) mod q)\tau(p) = 1 + ((p-1)\bmod q) described in Section 4? Doesn't a 192-token contiguous schedule mean short sequences never access row types 1, 2, and 3?

3.Have you conducted any mechanistic probing (e.g., via SAEs or per-block activation norms/ranks) to understand what FFN capabilities are lost when off-diagonal cycle products are removed?

Reviewer_aN5EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes replacing standard dense matrix multiplication inside transformer projections with a different learned bilinear product derived from finite-dimensional associative unital algebras. The main idea is to keep the same number of learned weight parameters while changing the fixed interaction table between activation blocks and weight blocks, thereby reducing arithmetic cost. On the modeling side, the authors lift these algebraic products to rectangular transformer projections using row types determined by token position, and argue compatibility with causal masking and KV-cached autoregressive decoding. They derive forward/training cost formulas and discuss finite-shape GPU constraints. Empirically, they benchmark custom kernels on representative transformer FFN shapes and report speedups under some regimes. They also train two approx. 110 M-parameter decoder-only language models from scratch under a matched recipe. The paper positions this as a feasibility result showing that such algebraic layers are trainable and can yield practical inference speedups, albeit with a quality trade-off at the tested scale.

Critical strengths

The shift from accelerating standard matrix multiplication to changing the multiplication law itself - while keeping the full parameter budget - is conceptually original and could open a meaningful new line of work. This matters to my assessment because novelty is often overstated in efficiency papers.

Critical weaknesses
  • The main training study uses a single small model scale, one run per condition, and modifies only FFN projections. Since quality degrades across all reported task metrics, it remains unclear whether the method is promising as a practical architecture or only as a proof of concept.

  • The paper reports end - to - end generation throughput, which is valuable, but also notes that response lengths differ between models. Since the method affects generation behavior, throughput measured on realized outputs may not isolate the layer - level efficiency gain. Moreover, since attention and LM head remain dense, the observed gain is only for a restricted insertion point. This matters because the central systems claim is about realized speedups, and the current setup does not cleanly separate algorithmic benefit from behavioral differences.

Questions for authors
  • Can the authors provide a controlled generation-throughput comparison with fixed prompt sets and fixed generated token counts per sample?

  • Can the authors clarify whether the row - type schedule introduces any positional artifacts measurable in ablations? For example, is cyclic assignment meaningfully different from blockwise assignment in language-model quality or optimization stability?

Reviewer_3UkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper changes the multiplication rule used by learned projections, retaining the weight bank while removing selected block interactions. It develops an associative-algebra construction and tests an algebraic FFN in a small language model.
Critical strengths
  • The distinction between changing a product and accelerating the same product is interesting. The connection to bilinear-rank bounds gives the construction a clear mathematical motivation, and the paper is candid about the observed quality loss.
Critical weaknesses
  • The paper does not demonstrate a quality-efficiency advantage over simpler alternatives. The roughly 7% generation-throughput gain comes with lower scores on every reported downstream metric. Without a smaller dense baseline or a structured/sparse projection at comparable compute, it is unclear why this trade-off is preferable. Retaining the same parameter count does not resolve this issue, because the function class and per-token interactions have changed.

  • The end-to-end evidence is limited to one training run per model, and the generated response lengths differ. These timings do not cleanly isolate the benefit of the algebraic layer; fixed-length decoding and repeated measurements are needed. The mathematical construction is interesting, but the current experiment establishes trainability rather than a compelling language-model advantage. I recommend weak rejection.

Questions for authors
  • Does requiring associativity improve the quality-speed trade-off compared with a similarly sparse block-interaction rule without that constraint? This would help explain what the algebraic structure contributes beyond reducing interactions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.