- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents an algebraic framework for describing neural operations, including convolution, linear attention, and Mamba, using elementary bilinear product operators and their compositions. Input and intermediate feature representations are modeled as elements of the tensor product of a position algebra, which determines positional routing, and a feature algebra, which determines feature interactions. The authors illustrate convolutional, local–local, and local–global routing, together with matrix products, learned structure-constant products, Clifford products, and Clebsch–Gordan products for SO(3) representations. They show how elementary operators can be composed to construct more complex neural blocks. Their architecture-generation recipe holds positional routing and operator composition fixed while changing the feature algebra. They evaluate three resulting variants: a combination of structure-constant and Clifford interactions in a linear-attention variant of ResNet-20 on CIFAR-10; a structure-constant state-space layer replacing the Mamba dynamical layer within Vision Mamba, also on CIFAR-10; and a Clebsch–Gordan interaction model for the QM9 missing-point task, retaining the routing and composition of linear attention while changing its feature interactions.
Critical strengths
- A useful separation of architectural design choices. Distinguishing positional routing from feature interactions clarifies relationships between architectures and enables controlled modifications to their computational structure.
- A systematic approach to architecture construction. Expressing familiar neural blocks as compositions of elementary operators provides a clear recipe for constructing architectural variants.
- A representative range of concrete algebraic examples. The paper illustrates several position and feature algebras that capture familiar neural operations and support alternative interaction mechanisms.
- Substantial improvements on the QM9 missing-point task. The Clebsch–Gordan model achieves considerably higher prediction accuracy and converges in fewer epochs than the matrix-interaction baseline.
Critical weaknesses
-
Clarity and presentation.
- I believe the manuscript requires substantial revision to make its central ideas accessible to readers unfamiliar with the algebraic formalism. The abstract and introduction introduce technical terminology before sufficiently motivating the construction or explaining its practical significance. The authors should provide more intuition about why each algebra is useful, which inductive biases it encodes, and how changing the feature algebra affects computation. A running example, a diagram separating positional routing from feature interactions, and simpler notation in the main text would improve readability without sacrificing technical rigor.
- Manuscript accuracy and submission requirements. Several concrete issues require correction. The provided manuscript appears to omit the AI-use statement required by the ICLR 2027 author guidelines. In addition, the bibliography incorrectly attributes Categorical Deep Learning to Bartosz Gavranović, David Spivak, Jacob Lurie, and John Baez. The authors should replace this entry with the verified author list and carefully check the remaining references. Correct bibliographic record.
- Placement of related work. Placing the substantive related-work discussion entirely in the appendix makes it difficult to assess the contribution from the main text. This is particularly problematic given the close relationship to Dong et al. The authors should include a concise comparison with the closest prior work in the main text, while retaining broader literature discussion in the appendix.
-
Novelty. Appendix A distinguishes the present framework from Dong et al. by introducing separate algebras for positional routing and feature interaction. However, the earlier work already introduces a feature algebra, an auxiliary structural algebra encompassing positional routing, and their tensor product. It also discusses fixed and learned structure constants, derives convolutional and SO(3)-equivariant operations, and analyzes attention and Mamba. Consequently, the manuscript does not clearly establish which aspects of its formalism are new. The authors should explicitly distinguish inherited constructions from new theoretical developments, architecture variants, and empirical findings.
-
Empirical evaluation.
- Limited comparison on the QM9 missing-point task. The strongest empirical improvements are reported on QM9, but the principal comparator is a matrix-interaction baseline that does not preserve the same rotational symmetry. This comparison supports the benefit of symmetry-preserving feature interactions, but does not establish an advantage over existing equivariant methods. Comparisons with established equivariant models under comparable training conditions would help determine whether the proposed construction offers benefits beyond enforcing a known symmetry constraint.
- Strength of the CIFAR-10 baselines. The reported ResNet-20 baseline achieves 89.67% accuracy, below the 91.25% as reported in Table 6 of the original ResNet paper. Although this does not invalidate the within-paper comparison, the discrepancy raises concerns about baseline strength and the robustness of the reported improvements. According to appendix D.1 the authors used 72×72, whereas the original evaluation uses 32×32 images and a different optimization protocol. The authors should explain the discrepancy and establish whether the improvements persist with well-tuned baselines under a standard CIFAR-10 protocol, using comparable tuning effort across methods.
Questions for authors
- In the linear attention the authors ignored the normalization term. Please clarify whether the feature transformation and normalization of linear attention are omitted only from the algebraic presentation or also from the experimental baseline, and explain how these components fit into the proposed framework.
- The experiments retain residual connections, but the formalism does not explicitly describe skip paths and additive branch merging. Clarifying these operations and their compatibility requirements would make the architecture-generation framework more complete.
Dear Reviewers,
As the discussion period approaches its conclusion, we would like to sincerely thank all reviewers for their positive feedback, and the opportunity to further clarify and strengthen our work. We remain very happy to address any outstanding questions or concerns.
We would also like to offer a final perspective on the broader significance of our contribution.
The central contribution of this work is not another neural architecture, but a mathematical framework for constructing neural architectures. Rather than treating convolution, attention, state-space models, and equivariant operations as separate architectural inventions, our framework reveals how they can be understood and systematically constructed through explicit algebraic choices.
This represents a shift in perspective: from designing neural operations primarily through empirical intuition to deriving architectural possibilities from mathematical structure. By separating positional routing, feature interaction, and operator composition, we turn the underlying algebra into an explicit and controllable architectural design variable.
Importantly, this is not merely a conceptual reformulation. Our new Clifford-SSM experiments demonstrate that changing the feature algebra can produce different recurrent-state representations, substantially reduce recurrent computation, and improve accuracy–efficiency trade-offs against reduced-state Mamba baselines. Together with our CIFAR-10 and QM9 experiments, these results demonstrate concrete consequences of the proposed construction principle.
We believe the significance of this contribution extends beyond the particular architectures and benchmarks evaluated here. The value of the framework lies not only in the architectures it produces today, but in the systematic exploration of architectural possibilities it enables for the future.
Our broader ambition is to open a new research direction in deep learning: one in which neural architectures are not only discovered through empirical experimentation, but systematically constructed, analyzed, and generalized through mathematical principles. This perspective has the potential to change how the community approaches architecture design, transforming algebraic structure from a tool for describing existing networks into a foundation for discovering new ones.
We hope this work will therefore be evaluated not simply as another architectural variant, but as a contribution toward a more systematic mathematical approach to neural architecture discovery and design.
We are grateful for the reviewers' suggestions, which have helped us improve both the clarity and the experimental validation of this work. We remain available and would welcome any further questions or discussion before the discussion period concludes.