acceptodds
Under review as a conference paper at ICLR 2027

Product Spectrum Neurons: Making Interaction Order Addressable

Abstract

Neural networks multiply features everywhere, yet interaction order usually remains entangled with width, depth, and gating rather than freely addressable. The Product Spectrum Neuron (PSNeuron) computes, in each of its \(R\) learned bases, elementary symmetric polynomials of degree \(2\) through \(K\) over \(G\) disjoint slots. Every order becomes its own channel, with three exact properties: on Rademacher inputs channel \(j\) carries Walsh degree \(j\); increasing \(K\) at fixed geometry appends orders without disturbing lower ones; and any order can be masked in a trained model without retraining or modifying learned parameters. Across three tabular tasks, one common PSNeuron architecture with \(K=4\) fixed in advance, a validation-selected layout, and task-specific learning rates removes 34.7% to 51.7% of the trainable parameters of the selected dense references while remaining within 0.44 points of their accuracy. At \(K=2\), narrow DINOv3 heads outperform an exact-parameter additive control in 69 of 75 paired seeds across 15 dataset and width settings, and on 20 fresh seeds a \(K=4\) DeltaNet mixer beats an exact same-slot additive twin with 48 fewer order coefficients and a conditional GELU control on associative recall after Holm correction. On controlled teachers, exact PSNeuron witnesses establish representability independently of training, order 6 content is recovered reliably near its target degree, and same-model spectra and masks verify the planted decomposition in recovered models and localize missing upper-order content in failed runs. PSNeuron turns interaction order into an addressable experimental coordinate of a neural site whose representability, recovery, and functional use can be examined within the same trained model.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Product Spectrum Neurons, interaction order, higher-order interactions, elementary symmetric polynomials, parameter efficiency, mechanistic interpretability, neural architectures

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sC8vOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have no idea what this paper is about.

Critical strengths

None

Critical weaknesses

As someone who is not from this subfield, I am largely unable to understand the current paper as written. The heavy use of jargon and technical language renders this paper inaccessible to non-experts to the point where I cannot even figure out what the problem being addressed is. My suggestion is to rewrite the abstract and introduction to give a more accessible introduction to what the problem is and how PSNeuron plans to solve this problem.

Questions for authors
  1. Can you explain at a high level what the problem this paper addresses is, and how PSNeuron plans to solve this problem?
  2. Can you define the jargon used in the paper, including “addressable”, “interaction order”, “structured interactions”, “dense capacity”, “site”, etc?
1 comment
AuthorsOpenReview ↗

Response to Reviewer sC8v

Thank you for emphasizing that the paper must explain its problem and terminology without assuming familiarity with this subfield. We rewrote the abstract and introduction to state the problem first, then explain PSNeuron through a worked example before formal notation (revised Sec. 1, pp. 1–2; Sec. 3, p. 3; Fig. 1, p. 4).

  1. The problem and approach. Many predictions depend on feature combinations even when no individual feature is informative. For example, if the label indicates whether two independent fair coin flips agree, knowing either flip alone does not predict it. Encoding heads as +1+1 and tails as −1-1, multiplying their values gives the answer. Neural networks already multiply features, but the degrees of these interactions are often not available as separately inspectable or removable channels. We ask whether a small replacement module can make such combinations explicit, retain predictive quality with fewer weights in some settings, and reveal which interaction orders a trained module uses.

PSNeuron learns weighted linear summaries of fixed groups of input features, called slots. Given three summaries a,b,ca,b,c, one channel computes ab+ac+bcab+ac+bc and another abcabc. A learned coefficient weights each channel, and a linear readout maps multiple sets of summaries to the output. Both channels reuse the summary weights, and either order can be masked after training without altering the other learned parameters. The worked example and Figure 1 now show how the calculation proceeds (Sec. 3, pp. 3–4). Reusing weights is expected to help when useful orders need common feature directions and can limit the model when they need different ones.

The original submission included two examples of where this construction helps. In a recurrent DeltaNet model on MQAR, a task requiring recall of values paired with queried keys, PSNeuron improves mean recall by 44.08 percentage points over an additive twin and 37.38 over a conditional GELU control. Both controls receive the same input information at nearly equal parameter budgets; the two differences survive Holm correction across 20 fresh seeds. Results vary: eight PSNeuron seeds reach at least 90% recall and six stay below 50% (Sec. 5.1; Table 58, p. 83). This does not imply reliable success on every seed or general superiority over every alternative. Narrow K=2K=2 heads on frozen DINOv3 image features also outperform equal-size additive heads in 69/75 paired seed comparisons, with a reversal at larger widths (Sec. 5.1; Tables 53, 55).

On three tabular datasets, a common K=4K=4 configuration removes 34.7–51.7% of selected dense reference models' trainable parameters while mean accuracy remains within 0.44 percentage points. This is a saving relative to those references, not an established improvement over the dense accuracy–size frontier (Sec. 5.2). Conversely, in the transformer attention-output MQAR setting, an identity carrier alone nearly solves the task; we explicitly state that this placement provides no evidence of a product advantage (Sec. 5.4).

  1. Definitions. Interaction order means the number of distinct factors multiplied: abab has order two and abcabc order three. Here the factors are learned summaries. Addressable means that an order channel can be inspected or disabled after training while parameters and other channels remain fixed. A site is the location of the replaced layer, and the host is the surrounding network (Introduction, p. 1). Exact polynomial degree is defined relative to the coordinates entering the module; a preceding nonlinear transformation can change its relation to raw features (Sec. 3, p. 3).

"Structured interactions" previously meant products taking one summary from each participating group, and "dense capacity" referred to the weights in the dense layer being replaced. The revised paper avoids these unexplained phrases and describes the products and parameter counts directly.

Representability means suitable weights exist to compute a target. Recovery measures whether training reaches a stated fit threshold; it does not require finding the exact constructed weights. Functional use is evaluated by switching an order off and measuring the effect on predictions. For controlled targets, order-six recovery succeeds in all 120 runs per setting with maximum order six or seven, while order-eight recovery succeeds in only 25% despite sufficient capacity. Masks then indicate which orders learned models use (Sec. 5.3; Table 23).

We hope these revisions clarify the problem, mechanism, and limits of the evidence, and would appreciate your reassessment in light of the revised manuscript.

Reviewer_yLYMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces an architecture that explicitly models and controls higher-order feature interactions through multiplicative operations. The authors evaluate the approach across tabular, vision, sequence, and synthetic tasks. Synthetic teacher and order-masking experiments demonstrate the model's ability to capture and utilize higher-order interactions. Parameter-matched comparisons with additive baselines further suggest that multiplicative interactions can improve predictive performance in certain settings.

Critical strengths

The paper introduces a well-motivated architecture that explicitly models and controls higher-order feature interactions. The authors provide a comprehensive empirical evaluation spanning tabular, vision, sequence, and synthetic tasks. Notably, the synthetic teacher experiments demonstrate the model’s capacity to capture and recover higher-order interactions, while the order-masking experiments further suggest that the learned models actively leverage these interactions. Moreover, the parameter-matched comparisons against additive baselines using DINOv3 features and DeltaNet offer compelling evidence that multiplicative interactions can yield predictive performance gains in certain settings.

Critical weaknesses

(1) The paper presents an extensive set of experiments across tabular classification, vision models, recurrent architectures, language modeling, and synthetic tasks. While this demonstrates the applicability of PSNeuron to different architectures, the experimental breadth comes at the expense of a systematic understanding of when and why PSNeuron is effective. Many experiments are conducted under relatively narrow architectural or training settings, making it difficult to isolate the factors responsible for the observed improvements.

Although the paper includes several controlled comparisons, the results remain fragmented across different tasks and configurations. A more focused analysis within a smaller number of representative settings would provide stronger evidence for the effectiveness of the proposed architecture.

(2) A key contribution is the explicit control of interaction order. However, the paper provides limited systematic analysis of how different values of (K) affect performance on real-world tasks. Although synthetic recovery and masking experiments are included, controlled comparisons across interaction orders would better demonstrate their practical benefits.

(3) PSNeuron combines multiple components, including grouped slots, shared factor banks, higher-order interactions, and order-wise calibration. While some ablations are provided, their individual contributions to performance remain unclear. More systematic comparisons against existing polynomial and higher-order interaction models would help establish the architectural advantages of PSNeuron.

(4) While the paper demonstrates the benefits of multiplicative interactions through comparisons with additive baselines, it lacks a systematic comparison with existing architectures that explicitly model higher-order feature interactions. Such comparisons would help clarify whether the observed improvements stem from modeling higher-order interactions in general or from the specific architectural design proposed in this work. Including these baselines would provide stronger evidence of the proposed method's advantages over existing approaches.

Questions for authors

Have the authors evaluated the proposed method on tasks such as Parity Check, NAE, and PINNs, where higher-order interactions may play an important role? If so, how does its performance compare with other architectures specifically designed to capture higher-order interactions?

1 comment
AuthorsOpenReview ↗

Response to Reviewer yLYM

Thank you for asking when PSNeuron helps and for requesting systematic order, component, and higher-order comparisons. We reorganized Results and added a retrospective analysis of archived CIFAR100 runs. No new model training was performed for the revision.

  1. When and why it helps. In the DeltaNet MQAR holdout (20 fresh seeds), PSNeuron improves mean recall by 44.08 points over an additive twin and 37.38 over conditional GELU; both survive Holm correction (p=0.0020p=0.0020 and p=0.0027p=0.0027). These controls read the same state/token inputs, at closely matched site counts (4,352 versus 4,400 parameters). Eight PSNeuron seeds solve MQAR and six stall; the SwiGLU comparison is unresolved. This supports the complete construction, not any single component (Sec. 5.1; Table 58, p. 83). On frozen DINOv3 features, narrow K=2K=2 heads lead equal-size additive heads in 69/75 paired seeds, with eight of 15 corrected setting-level comparisons, but grid-edge tuning and reversal at larger widths limit the finding (Tables 53 and 55). Common K=4K=4 tabular replacements save 34.7–51.7% of selected dense references' whole-model parameters with mean accuracy within 0.44 points, without demonstrating equivalence or dense-frontier superiority (Sec. 5.2).

  2. Real-task interaction-order sensitivity. The original standalone sweep compares K=2,4,6,8K=2,4,6,8 on four tasks, five seeds each. K=4K=4 has the highest mean on three tasks and K=2K=2 on EuroSAT; budgets vary and all selected learning rates hit grid boundaries, so no optimal KK follows (Table 43, p. 68). We additionally analyze seven archived CIFAR100 capacity groups with paired seeds, compatible protocols, interior rates, and at most 1.44% within-group site-count spread. In the fixed-allocation group (R=G=8R=G=8, readout rank 8), accuracies are 39.49/39.73/39.77/39.78% for K=2/4/6/8K=2/4/6/8 at site counts 6,144/6,168/6,184/6,200. Other groups give different order rankings; in five further complete but protocol-ineligible groups, K=8K=8 scores below K=2K=2. This descriptive, retrospectively defined comparison does not isolate KK or identify an optimum (Table 51, p. 75; Sec. J.10). Equal-budget DINOv3 controls restricted to order 3, order 4, or a fixed 2–4 mixture have lower means than the K=2K=2 head at width 3 across five datasets; these are not learned K=4K=4 spectra (Table 54, p. 78).

  3. Component controls. The additive twin shares conditioners, bank, slots, and readout at paired initialization but replaces products by summed GELUs; conditional GELU preserves conditioner shape and input access, but uses dense aggregation. Neither comparison isolates higher orders, weighting, or calibration. DINOv3 at K=2K=2 addresses pairwise products only. At exactly equal parameter counts, a custom HOFM-style singleton control yields no resolved grouping advantage on three tabular tasks (Table 61, p. 84). Synthetic factor-sharing targets favor shared banks when orders reuse directions (separate banks need 2.00–2.25 times the factors) and separate banks when directions differ (shared needs 3.43 times more); these are exploratory width ratios, not real-task ablations (Table 27, p. 56). Removing the conditioner also removes 59–66% of parameters, so that ablation mixes architecture and budget. Calibration's independent effect remains untested (Table 47; Sec. 6).

  4. Higher-order models. Generic bilinear heads on DINOv3 features are not exactly parameter matched. PSNeuron has higher means than the nearest smaller heads in all 15 settings (12 corrected intervals); against the nearest larger heads, nominal intervals favor each family in six settings. With the same number of product units, bilinear heads use 83–98% more parameters and have higher means in all ten settings (three corrected intervals; Table 56, p. 80). Thus these controls do not isolate a grouping-specific benefit. Related Work credits HOFM with ANOVA interactions, degree weights, factor sharing, and dynamic programming; our contribution is a grouped vector-valued replacement module, same-model order interventions, and separate tests of representability, recovery, and use. We do not claim to have invented degree weighting. A trained benchmark against published higher-order methods remains open (Secs. 2 and 6).

  5. Parity, NAE, and PINNs. A K=8K=8 unit fits all 256 eight-bit parity inputs at 99.4±1.499.4\pm1.4% (seed standard deviation), versus 52.1% additive and 76.5% GELU. This tests fitting the full truth table, not unseen-input generalization. Held-out XOR probes examine order control; a larger GELU can also succeed (Table 41, p. 67). We have not evaluated NAE, PINNs, or specialized higher-order baselines and make no claims for them.

Reviewer_aK3QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces PSNeuron, which aims to replace standard dense neural network components with explicitly computed polynomial interactions. This separation of interactions into specific channels aims to improve the interpretability of model function, while also improving efficiency via the reduction in parameters. The authors test PSNeuron across a variety of architectures and datasets, finding improvements in certain configurations.

Critical strengths

The paper includes extensive experimental results across a variety of architectures and tasks, and tackles the problem of interpretability in terms of understanding how features interact.

Critical weaknesses

Unfortunately, the paper is incredibly challenging to parse. The language is overly dense and the prose structure makes the paper very difficult to understand: the way the paper is written assumes that the reader is deeply familiar with every concept before properly establishing those concepts. For example, just in the introduction: what is interaction order? Why does it shift with width, depth, and gating? What is a site? What is a learned linear slot? What is a channel? What is a grouped product vs. an additive head? These concepts being immediately introduced essentially without any buildup is very jarring.

The same issue renders the contributions very difficult to understand. While the experiments included in the paper span a variety of tasks and model architectures, it is arduous to piece together how the individual results, such as parameter efficiency, PSNeuron replacement quality, and which configurations benefit, build up to the wider narrative. For example, the MQAR results at line 315 seem to suggest that PSNeuron is not necessary compared to an identity carrier, thus leaving unclear what this result demonstrates about the benefits of the technique. Ultimately, the results feel more like a collection of disparate findings without a clear explanation of how they fit together.

Overall, the current state of the paper makes it challenging to identify a clear hypothesis and a rigorously supported scientific contribution even after several readings. The work would benefit substantially from a revision, including improvements in exposition structure, a more cohesive interpretation of the results, and a clearer scientific narrative.

Questions for authors

No further questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aK3Q

Thank you for identifying how the presentation obscured the central question and the connections among our experiments. We rewrote the abstract and introduction and reorganized Results around whether products of shared feature summaries support compact prediction while exposing the interaction orders used by a trained module. No models were trained for this revision; we clarified existing experiments and retrospectively analyzed archived runs.

Computation and motivation. We now introduce order before notation: x1x2x_1x_2 combines two factors and x1x2x3x_1x_2x_3 combines three. A slot is a learned linear summary of a fixed input group. For slots a,b,ca,b,c, PSNeuron computes the pairwise channel ab+ac+bcab+ac+bc and the third-order channel abcabc; both reuse the summaries, and a linear readout produces the output. Figure 1 walks through the calculation (revised Secs. 1 and 3; Fig. 1, pp. 1, 3–4). A site is the location of a replaced module; the host is its surrounding network. An additive control sums GELU-transformed slots instead of multiplying them. Width alone does not increase polynomial degree; gating and composition may mix orders, while PSNeuron exposes separate degrees relative to its input coordinates. Sharing can save parameters when orders reuse feature directions and restrict recovery when directions differ (Table 27; Sec. 5.4).

How the evidence fits together. The Results distinguish four contributions: (i) exact, maskable order channels; (ii) predictive effects of product aggregation; (iii) parameter savings relative to named dense references; and (iv) representability, recovery, and functional use. A common K=4K=4 tabular replacement removes 34.7–51.7% of selected dense references' whole-model parameters while mean accuracy stays within 0.44 points; this does not prove equivalence or a better dense accuracy–size frontier (Sec. 5.2). Exact parameter constructions establish representability of controlled teachers. Training recovers order-6 targets in all 120 runs per setting for maximum order 6 or 7, but order-8 targets in only 25% despite sufficient capacity. Masks show which orders contribute and which upper-order content is missing after failed recovery (Sec. 5.3; Table 23).

MQAR and the identity carrier. We agree with your reading of original line 315. At the transformer attention-output site, removing the 16,512-parameter projection and retaining only the identity carrier already gives 99.96% recall; PSNeuron gives 99.97%, and an equal-size linear correction 99.98%. Thus the projection is dispensable there: this is not evidence that products help at that site (original Table 30, p. 55; revised Table 30, p. 58).

The positive MQAR evidence concerns a different site and host: DeltaNet's recurrent channel mixer. The original submission already contained a holdout of 20 fresh seeds. Its primary controls use the same state/token inputs at nearly equal site budgets: 4,400 PSNeuron parameters versus 4,352 for the additive twin and conditional GELU. PSNeuron improves mean recall by 44.08 and 37.38 percentage points, respectively (Holm-adjusted p=0.0020p=0.0020 and p=0.0027p=0.0027). Figure 2 now shows all seed outcomes and paired differences (Sec. 5.1; Table 58, p. 83). These gains support the complete product-spectrum construction, not an isolated effect of higher orders, order weighting, or calibration. Outcomes vary: 8/20 PSNeuron seeds solve the task and 6 stall; SwiGLU remains unresolved.

As complementary evidence without recurrent dynamics, narrow K=2K=2 heads on frozen DINOv3 features lead equal-size additive heads in 69/75 paired seed comparisons; eight of 15 setting-level comparisons survive Bonferroni correction. Grid-edge learning rates qualify this result, and the advantage reverses for wider heads (Tables 53 and 55). We hope the revised structure makes both the contribution and its limits clearer.

I would like to thank the authors for their rebuttal, and appreciate the explanation of the motivation, the discussion of the results, and the clarification of MQAR and the identity carrier. However, such changes drastically alter the fundamental structure of the paper itself. Indeed, when reviewing the revised manuscript, I noticed that a substantial portion of the prose in the main text has been modified. While I commend the effort, I believe that such drastic changes require a new cycle of reviews to properly assess its updated scientific narrative and contribution. I thus maintain my score and assessment of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.