acceptodds
Withdrawn from ICLR 2027

PolyGate: Continuous Power-Product Gating for Deep Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Standard feedforward and convolutional neural networks parameterized with piecewise-linear activations approximate smooth polynomial and multiplicative interactions through piecewise-linear tessellations, whereas multiplicative architectures can represent power-law and cross-variable interactions directly. However, historical product-unit formulations suffered from unstable optimization, logarithmic singularities near the origin, and undefined real-valued fractional powers on negative inputs. In this work, we propose PolyGate, an architectural primitive that augments affine coordinate bases with learnable continuous power-product gating. To ensure well-defined real-valued power operations and exclude zero-boundary logarithmic singularities from the operating domain, we introduce Strict-Positivity Batch Normalization (SP-BN) and explicit positive-domain handling. We further introduce dimension-normalized multiplicative gating to control activation variance as feature dimension grows. Theoretically, we characterize exact representation of univariate and multivariate generalized monomials, gradient regularity on strictly positive bounded domains, near-identity initialization with asymptotic recovery of the affine baseline, and dimension-scaling laws of multiplicative gates. Empirically, PolyGate reduces test MSE by on Feynman relativistic mechanics, achieves higher mean/median out-of-distribution across the complete 10-equation AI-Feynman suite, and is evaluated across convolutional backbones on Tiny-ImageNet-200: delivering to gains on edge MobileNetV4 with modest parameter overhead ( to ), and to gains on ConvNeXt with dual depthwise-pointwise gating.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neural network architectures, polynomial gating, multiplicative interactions, numerical stability, representation learning, scientific machine learning, computer vision, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PSctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes POLYGATE, a neural network architecture that combines affine transformations with learnable continuous power-product gates. The authors introduce positive-domain normalization, dimension-normalized gating, and a staged optimization strategy to improve stability. The method is evaluated on physical-law regression, synthetic monomial extrapolation, tabular datasets, and vision benchmarks. The results show improvements on some physical-law tasks and ConvNeXt models, while the benefits are less consistent on other tasks.

Critical strengths
  • The paper addresses a relevant problem in scientific machine learning. Explicitly modeling multiplicative interactions and power laws can provide a useful inductive bias for extrapolation.

  • The paper combines theoretical analysis with experiments across several domains. The integration of power-product gating into convolutional architectures is interesting, and the results on some physical-law and ConvNeXt tasks are promising.

Critical weaknesses
  • The novelty is limited compared to existing product units, Neural Power Units, and polynomial neural networks. The paper does not directly compare against the closest related methods, making it difficult to assess the advantage of the proposed combination.

  • The theoretical claims are not fully consistent with the implementation. In particular, the exact monomial representation results do not account for the numerical clamps used in the implemented operator.

  • The experimental reporting contains several inconsistencies. Table 2 specifies 10 epochs for the AI-Feynman experiments, whereas Table 6 specifies 4 warm-up and 4 specialization epochs for Feynman, and Figure 3 shows an 8-epoch run. In addition, the variance calculation in Appendix I gives approximately 675.3 for the stated settings, while the text claims a value above 1,000. Appendix H.3 makes a similar claim at rank 128 without specifying the parameters needed to verify it.

  • The empirical results are mixed. POLYGATE often obtains negative OOD (R^2) on the controlled monomial tasks and underperforms standard MLPs and tree-based methods on tabular datasets. The vision gains are modest in some settings, while the larger ConvNeXt improvements come with substantial parameter and latency overhead.

Questions for authors

1- How do the authors reconcile the exact monomial representation claims with the numerical clamps used in the implementation? Can they clarify which formulation was used in the experiments?

2- Why is POLYGATE not directly compared against existing product units, Neural Power Units, and polynomial neural networks under matched experimental settings?

3- Can the authors clarify the inconsistencies in the reported training budgets and variance calculations, and provide the exact configurations used for the main results?

4- Can the authors provide ablations to isolate the contributions of positive-domain normalization, geometric-mean gating, and DPBO?

Reviewer_x9y5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PolyGate, a gated neural network architecture designed to enhance the representation of higher-order and multiplicative interactions. The authors introduce a learnable power-product gating mechanism integrated with conventional affine transformations. They provide theoretical analyses of the proposed architecture, including its representational capacity, gradient properties, and initialization behavior. Furthermore, they conduct experiments on synthetic regression tasks, physical law extrapolation, and image classification benchmarks to evaluate the effectiveness of the proposed approach.

Critical strengths

One of the strengths of this paper is that it presents a simple and intuitive neural network architecture that directly addresses the identified limitations of conventional neural networks. In particular, the authors introduce a power-product gating mechanism to enhance the representation of higher-order feature interactions and multiplicative relationships.

Another strength is that the authors provide mathematical analyses to support the motivation and design of the proposed architecture. In addition, the paper presents optimization strategies for training the proposed model, which further strengthens the overall methodological contribution of the work.

Critical weaknesses

(1) One of the main weaknesses of this paper is the limited selection of baseline methods in the experimental evaluation. Although the authors compare the proposed method with standard MLPs, wider MLPs, and KANs, these comparisons are insufficient to establish its advantages over existing approaches designed to model higher-order nonlinearities and multiplicative feature interactions.

In particular, TMPNN (Ivanov & Ailuro, 2024), Volterra Neural Networks (Roheda et al., 2024), and TESLA (Ko et al., 2026) are relevant approaches. Although these methods employ different computational mechanisms, they share the goal of improving the representation of higher-order or nonlinear interactions. Direct comparisons with these methods, particularly on synthetic polynomial regression and physical law extrapolation tasks, would provide stronger empirical evidence for the effectiveness and novelty of PolyGate.

(2) The ConvNeXt-Small baseline in Table 5 exhibits unusually high variability, reporting an accuracy of 39.95 ± 8.78%. This raises concerns about the stability of the baseline training and whether the substantial improvements attributed to PolyGate are evaluated against sufficiently well-optimized baselines.

Furthermore, although PolyGate consistently outperforms the standard CNN baselines in Table 4, it underperforms the parameter-matched wider CNNs across all evaluated datasets. In addition, the substantial accuracy improvements reported for ConvNeXt are accompanied by nearly doubled parameter counts and considerably increased inference latency. These results call for a more careful assessment of the claimed representational and computational efficiency.

Questions for authors

Could the authors provide additional comparisons with these methods, such as TMPNN, Volterra Neural Networks, TESLA or any other methods, to better demonstrate the effectiveness and novelty of PolyGate?

Reviewer_zN1FOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PolyGate, which multiplies an affine carrier by a power-product gate with learnable exponents, z=(∑iwixi+b)∏i(xiγi+θi)z=(\sum_i w_ix_i+b)\prod_i(x_i^{\gamma_i}+\theta_i). Inputs are kept in [1,∞)[1,\infty) by Strict-Positivity BN, wide layers use a geometric-mean gate, and a two-stage scheme (DPBO) trains the carrier first and then only the gates. The paper proves exact monomial representation and several stability properties. Experiments cover OOD extrapolation on AI-Feynman equations and synthetic monomials, tabular regression, small CNNs on MNIST/CIFAR, and MobileNetV4/ConvNeXt on Tiny-ImageNet.

Critical strengths
  1. The paper gives concrete fixes for the known failure modes of product units: undefined powers for negative inputs, the log singularity at zero, and the lack of an identity baseline.

  2. On several multiplicative AI-Feynman laws, PolyGate extrapolates better than a parameter-matched MLP (e.g., drag OOD R2R^2 0.93 vs 0.57, Table 9).

  3. Code, 3-seed scripts and failure cases are reported openly.

Critical weaknesses
  1. PolyGate is worse than parameter-matched baselines. In Table 4, the matched-wider network has the same parameter count and beats PolyGate on all four datasets (e.g., CIFAR-100 63.65 vs 61.05). How does PolyGate compare with NPU, Π\Pi-nets or GLU/SE gates at matched FLOPs, and with an ablation that keeps SP-BN but removes the power gate?

  2. The extrapolation claim relies on a misleading metric. EER rewards a worse in-distribution fit: on synthetic monomials, PolyGate's OOD R2R^2 is worse than both MLPs in 11 of 12 settings (Table 11), contradicting Theorem 2. In addition, EQ-03 and EQ-04 have identical numbers in Table 9 but are counted as two wins. What are the OOD results under joint training, and why are these two rows identical?

  3. The gains lack proper controls. Iso-budget DPBO ends below the 20-epoch baseline on all datasets (Table 7). The ConvNeXt baselines appear not to have trained properly (Small 39.95$\pm$8.78, below MobileNetV4-Small). How large are the gains against properly trained ConvNeXt baselines, and against a baseline trained for the same extra epochs?

  4. The novelty is limited and the reporting is inconsistent. Learnable exponents (NPU) and polynomial networks (Π\Pi-nets) already exist, and most theorems follow directly by substitution. Training epochs, in-distribution R2R^2 and the 87.4% MSE claim disagree across Tables 2, 6 and 9 and Fig.~3a. Is the 87.4% reduction measured in or out of distribution?

Questions for authors

Please see the questions in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.