acceptodds
Under review as a conference paper at ICLR 2027

ALGEBRAIC INTERACTION: A RECIPE FOR NEURAL ARCHITECTURE GENERATION

Abstract

We introduce an algebraic foundation for neural architecture design in which convolution, attention, state-space models, and equivariant layers are expressed through compositions of bilinear product operators over two structures: a position algebra governing positional routing and a feature algebra governing feature interactions. This decomposition makes routing, feature algebra, and operator composition explicit design variables. We focus on the feature-algebra axis through an Algebraic Recipe: fix the routing and operator composition of an existing architecture while replacing its feature algebra. This preserves the information-flow structure while allowing computational and symmetry properties to be modified through algebraic specification. We instantiate the recipe in linear attention, Mamba, and equivariant networks. Structure-constant and Clifford algebras improve accuracy at comparable parameter count in linear attention; structure-constant algebras yield up to faster training and inference in Vision Mamba at competitive accuracy; and an -representation algebra provides rotational equivariance by construction. These results demonstrate that feature algebra alone can control accuracy, efficiency, and symmetry without redesigning positional routing. This turns algebraic structure into an explicit design variable and reduces architecture design, within our framework, to algebraic specification.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Neural architecture design, Algebraic structures, Feature interaction, State-space models, Attention, Equivariance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_krbYOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents an algebraic framework for describing neural operations, including convolution, linear attention, and Mamba, using elementary bilinear product operators and their compositions. Input and intermediate feature representations are modeled as elements of the tensor product of a position algebra, which determines positional routing, and a feature algebra, which determines feature interactions. The authors illustrate convolutional, local–local, and local–global routing, together with matrix products, learned structure-constant products, Clifford products, and Clebsch–Gordan products for SO(3) representations. They show how elementary operators can be composed to construct more complex neural blocks. Their architecture-generation recipe holds positional routing and operator composition fixed while changing the feature algebra. They evaluate three resulting variants: a combination of structure-constant and Clifford interactions in a linear-attention variant of ResNet-20 on CIFAR-10; a structure-constant state-space layer replacing the Mamba dynamical layer within Vision Mamba, also on CIFAR-10; and a Clebsch–Gordan interaction model for the QM9 missing-point task, retaining the routing and composition of linear attention while changing its feature interactions.

Critical strengths
  • A useful separation of architectural design choices. Distinguishing positional routing from feature interactions clarifies relationships between architectures and enables controlled modifications to their computational structure.
  • A systematic approach to architecture construction. Expressing familiar neural blocks as compositions of elementary operators provides a clear recipe for constructing architectural variants.
  • A representative range of concrete algebraic examples. The paper illustrates several position and feature algebras that capture familiar neural operations and support alternative interaction mechanisms.
  • Substantial improvements on the QM9 missing-point task. The Clebsch–Gordan model achieves considerably higher prediction accuracy and converges in fewer epochs than the matrix-interaction baseline.
Critical weaknesses
  • Clarity and presentation.

    • I believe the manuscript requires substantial revision to make its central ideas accessible to readers unfamiliar with the algebraic formalism. The abstract and introduction introduce technical terminology before sufficiently motivating the construction or explaining its practical significance. The authors should provide more intuition about why each algebra is useful, which inductive biases it encodes, and how changing the feature algebra affects computation. A running example, a diagram separating positional routing from feature interactions, and simpler notation in the main text would improve readability without sacrificing technical rigor.
    • Manuscript accuracy and submission requirements. Several concrete issues require correction. The provided manuscript appears to omit the AI-use statement required by the ICLR 2027 author guidelines. In addition, the bibliography incorrectly attributes Categorical Deep Learning to Bartosz Gavranović, David Spivak, Jacob Lurie, and John Baez. The authors should replace this entry with the verified author list and carefully check the remaining references. Correct bibliographic record.
    • Placement of related work. Placing the substantive related-work discussion entirely in the appendix makes it difficult to assess the contribution from the main text. This is particularly problematic given the close relationship to Dong et al. The authors should include a concise comparison with the closest prior work in the main text, while retaining broader literature discussion in the appendix.
  • Novelty. Appendix A distinguishes the present framework from Dong et al. by introducing separate algebras for positional routing and feature interaction. However, the earlier work already introduces a feature algebra, an auxiliary structural algebra encompassing positional routing, and their tensor product. It also discusses fixed and learned structure constants, derives convolutional and SO(3)-equivariant operations, and analyzes attention and Mamba. Consequently, the manuscript does not clearly establish which aspects of its formalism are new. The authors should explicitly distinguish inherited constructions from new theoretical developments, architecture variants, and empirical findings.

  • Empirical evaluation.

    • Limited comparison on the QM9 missing-point task. The strongest empirical improvements are reported on QM9, but the principal comparator is a matrix-interaction baseline that does not preserve the same rotational symmetry. This comparison supports the benefit of symmetry-preserving feature interactions, but does not establish an advantage over existing equivariant methods. Comparisons with established equivariant models under comparable training conditions would help determine whether the proposed construction offers benefits beyond enforcing a known symmetry constraint.
    • Strength of the CIFAR-10 baselines. The reported ResNet-20 baseline achieves 89.67% accuracy, below the 91.25% as reported in Table 6 of the original ResNet paper. Although this does not invalidate the within-paper comparison, the discrepancy raises concerns about baseline strength and the robustness of the reported improvements. According to appendix D.1 the authors used 72×72, whereas the original evaluation uses 32×32 images and a different optimization protocol. The authors should explain the discrepancy and establish whether the improvements persist with well-tuned baselines under a standard CIFAR-10 protocol, using comparable tuning effort across methods.
Questions for authors
  • In the linear attention the authors ignored the normalization term. Please clarify whether the feature transformation and normalization of linear attention are omitted only from the algebraic presentation or also from the experimental baseline, and explain how these components fit into the proposed framework.
  • The experiments retain residual connections, but the formalism does not explicitly describe skip paths and additive branch merging. Clarifying these operations and their compatibility requirements would make the architecture-generation framework more complete.
Reviewer_SeStOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents an approach to developing model architectures by using feature and position algebras. These algebras generalize the process of designing a neural network and allow for structures that are more efficient and effective in both training speeds and performance. They show on QM-9 and CIFAR-10 standard networks with layers replaced by the clifford architecture design of networks see the aforementioned improvements.

Critical strengths

In a world where large and slow models are especially prevalent, seeing this work as part of an effort to reduce parameter counts while increasing performance and speed is a welcome one. Their work presents a unique way of designing networks that generalizes to equivariant networks which can be especially promising in the design of equivariant networks with more complicated groups or symmetry breaking components. I believe their work should be accepted as it is incredibly novel and proposes a potential way of building networks more efficient and effective for data.

Critical weaknesses

While they show great improvements in training speed and performance (especially on QM9 and on the variants of MAMBA) the total performance difference on CIFAR-10 is fairly minimal. Additionally, the great jump in performance on QM9 may just be because of the non-optimal baseline chosen with it. I would like to see a model built purely from their method to see if that continues to up performance rather than just select layers, and I would like to see a better choice for the QM9 baseline. I worry their results may not be reflective to a full design from the ground up.

Questions for authors
  1. What would happen if instead of using the architectures built by the algebras in place of the layers in the selected neural networks, you used the standard design of those layers, how would that compare in the same performance?
  2. Why did you choose a matrix multiplication model to test against QM9 instead of another equivariant graph neural network?
Reviewer_mqUkOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an algebraic formalism that expresses mechanisms such as convolution, linear attention, state-space models, and equivariant layers through positional routing and feature interaction. Signals are represented in a tensor product of a position algebra and a feature algebra. The resulting "Algebraic Recipe" keeps routing and operator composition fixed while replacing the feature algebra. The authors instantiate this recipe through λ\lambda-Clifford interactions, λ\lambda-SSM, and SO(3)-compatible Clebsch-Gordan interactions. Experiments on CIFAR-10 and a QM9 missing-point prediction task report accuracy gains at comparable parameter counts, higher throughput at competitive accuracy, and benefits from symmetry-compatible interactions.

Critical strengths
  1. Conceptual unification: Separating positional routing from feature interaction provides a coherent, interpretable vocabulary for describing how different architectural mechanisms are constructed and related. Treating feature algebra as an explicit design variable is an appealing perspective that could help organize architectural choices.

  2. Concrete architectural instantiations: The framework goes beyond descriptive reformulation by producing implemented models. The λ\lambda-Clifford accuracy comparison, λ\lambda-SSM throughput measurements, and QM9 symmetry experiment demonstrate practical consequences of changing feature interactions in the tested settings.

Critical weaknesses
  1. Limited empirical support for generality and comparative efficiency: Evaluation is restricted to CIFAR-10 and a QM9 missing-point task, leaving the usefulness of the proposed swaps at larger model scales or longer sequence lengths uncertain. More specifically, the approximately 3×3\times throughput improvement in Table 2 accompanies a reduction in recurrent-state size from 64×16=102464\times16=1024 components to 16 components per head. The authors explicitly describe this reduction as a consequence of the algebra substitution. However, without a conventional Mamba baseline with reduced state size or comparable computational resources, the comparison does not establish whether the proposed interaction offers a better accuracy-efficiency tradeoff than ordinary state-size reduction. Table 2 also omits parameter counts and FLOPs.

  2. The distinct contribution relative to the closest algebraic formalism needs clarification: Appendix A contrasts the proposed framework with Dong et al. (2026), emphasizing more detailed interaction structure. However, the boundary between the present contribution and that work's algebraic representations, operator compositions, and architecture-generation principles remains insufficiently explicit. Since the main contribution is the framework rather than a single architecture, a clearer account of which definitions, construction principles, and empirical findings are new is important for assessing its significance.

Additional presentation feedback: A main-text schematic showing how routing and operator composition are preserved while feature interactions change would make the recipe easier to follow. The captions of Appendix Figures 1-4 could also provide more explanation of the comparisons.

Questions for authors
  1. Baseline fairness and implementation: How does λ\lambda-SSM compare with conventional Mamba variants using reduced recurrent-state sizes and comparable computational budgets? Please clarify parameter counts, FLOPs, and accuracy-throughput tradeoffs. How were both the proposed interactions and baseline Mamba scan implemented and optimized, including any fused or custom kernels and the timing protocol?

  2. Relation to Dong et al. (2026): Could you identify the relevant version of the cited work and explicitly distinguish the formal definitions, algebraic recipes, architecture constructions, and empirical findings that are new to this submission?

  3. Scale and generality: Are any results already available beyond CIFAR-10 and the QM9 missing-point task, particularly at larger model scales or longer sequence lengths? Such evidence would help assess how broadly the demonstrated benefits extend.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period approaches its conclusion, we would like to sincerely thank all reviewers for their positive feedback, and the opportunity to further clarify and strengthen our work. We remain very happy to address any outstanding questions or concerns.

We would also like to offer a final perspective on the broader significance of our contribution.

The central contribution of this work is not another neural architecture, but a mathematical framework for constructing neural architectures. Rather than treating convolution, attention, state-space models, and equivariant operations as separate architectural inventions, our framework reveals how they can be understood and systematically constructed through explicit algebraic choices.

This represents a shift in perspective: from designing neural operations primarily through empirical intuition to deriving architectural possibilities from mathematical structure. By separating positional routing, feature interaction, and operator composition, we turn the underlying algebra into an explicit and controllable architectural design variable.

Importantly, this is not merely a conceptual reformulation. Our new Clifford-SSM experiments demonstrate that changing the feature algebra can produce different recurrent-state representations, substantially reduce recurrent computation, and improve accuracy–efficiency trade-offs against reduced-state Mamba baselines. Together with our CIFAR-10 and QM9 experiments, these results demonstrate concrete consequences of the proposed construction principle.

We believe the significance of this contribution extends beyond the particular architectures and benchmarks evaluated here. The value of the framework lies not only in the architectures it produces today, but in the systematic exploration of architectural possibilities it enables for the future.

Our broader ambition is to open a new research direction in deep learning: one in which neural architectures are not only discovered through empirical experimentation, but systematically constructed, analyzed, and generalized through mathematical principles. This perspective has the potential to change how the community approaches architecture design, transforming algebraic structure from a tool for describing existing networks into a foundation for discovering new ones.

We hope this work will therefore be evaluated not simply as another architectural variant, but as a contribution toward a more systematic mathematical approach to neural architecture discovery and design.

We are grateful for the reviewers' suggestions, which have helped us improve both the clarity and the experimental validation of this work. We remain available and would welcome any further questions or discussion before the discussion period concludes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.